For schema evolution Mergeschema can be used in Spark for Parquet file formats, and I have below clarifications on this
Does this support only Parquet file format or any other file formats like csv,txt files.
If new additional columns are added in between I understand Mergeschema will move the columns to last.
And if column orders are disturbed then whether Mergeschema will align the columns to correct order when it was created or do we need to do this manually by selecting all the columns.
Update from Comment :
for example If I have a schema as below and create table as below –
spark.sql("CREATE TABLE emp USING DELTA LOCATION '****'")
empid,empname,salary====> 001,ABC,10000 and next day if I get below format
Whether new columns –
empage, empdept will be added after