Upload formats

What the platform reads, where that is decided, and how to add one.

1. The one list

UploadFormat is the authority. It lives in common because all three applications need it and that is the only module all three depend on. Everything else reads it:

This list used to exist six times and had already drifted: .xls was offered by the form and read by nothing, the draft panel left it out, the preview left .tsv out, and the Power BI truncator cut a .txt the parser refused. Do not add a seventh copy. A format is added once, here.

Each constant carries what it can do, not just its name:

• text() — its rows are lines, so it can be counted and cut without decoding, and a Power BI artifact can carry it verbatim inside Csv.Document.
• compressed() — delimited text behind a gzip wrapper. The engine inflates it, bounded while inflating, before any parser sees it.
• renderable() — a spreadsheet. Not text, but one sheet of it becomes text in process: see §3.
• bundle() — an Apache Superset export, which describes a dataset and carries no rows at all.

2. What is accepted

Not planned, and why: Google Sheets live (the platform never calls an online service — export to .xlsx), Apple Numbers (no Java reader), .sav/.dta/.sas7bdat (heavy readers, no demand), .sqlite/.accdb (a database is an Apache Superset connection, not an upload), Arrow/Feather/Avro (Parquet covers the columnar case). .xlsb is not implemented: see §7.

3. Spreadsheets

Which sheet. Hidden and very-hidden sheets are skipped. Among the visible ones, the sheet with the most data rows under a header wins; a tie goes to workbook order. The winner's name is recorded on DatasetSchema.sourceSheet and travels with everything built from it — a rebuilt HTML version reads the sheet the first version read, and Power Query names that sheet rather than taking whatever is first in the navigation table.

A caller may override it with the sheet parameter, on every upload endpoint. A name that is not there answers 400 sheet-not-found listing the visible sheets, before a job row exists and before studio charges. POST /datasets/inspect returns the sheets, their headers, their row counts and the one that would be elected; it runs no agent and stores nothing, so it is free and is on the guest surface too.

Which row is the header. Within the first 20 rows, the first row where at least two cells are filled, at least half of them are text rather than numbers, and the next non-blank row carries a number. That skips the title line — even a two-cell one, whose next row is the header and has no number in it — and the blank spacer an export so often starts with. A table that is text all the way down has no number to find, so the older rule stands behind it: the first row of at least two cells with anything under it. The rule lives in CellValues.headerRow, and all three readers hold the first 40 rows until it has decided, so a workbook of 5 MB and one of 50 MB find the same row. Rule-based on purpose: asking the agent would turn every upload into a metered run.

Workbooks: the sheet and the header row, chosen (T13). Both can be named on every upload endpoint and on POST /datasets/inspect: sheet by name, headerRow as the 0-based row of that sheet. Either is ignored by a file that has no sheets, so one request covers a mixed set. inspect answers each sheet's guessed headerRow, the file's headerRowGuess, and a preview — the named sheet (else the elected one) read under the named row (else the guess): its columns, its first ten rows and its row count. The parse card in SourcePanel.tsx is built on that: a sheet dropdown and a header-row stepper, each change asking again. A named row that is blank answers 422 unreadable-file in words, and the generate calls check it before a job row exists, at the cost of a second read of the workbook only when a row was named. The row that was read is recorded on DatasetSchema.sourceHeaderRow beside sourceSheet, for the sheet's reason: an HTML rebuild (html_source/html_revision.source_header_row, generator V018) must read the rows the first version read, and a linked Power BI workbook skips to the header before promoting it — by content, up to the first row holding the first column's name, because Power Query's sheet table starts at the used range, not at the row the parser counted from.

Header cells are read by column index, not by POI's row iterator, which skips cells the row never defined. A gap becomes column_C rather than shifting everything after it one place left, and a numeric header (2024) is a name rather than an exception.

Cell values are decided once, in engine-core CellValues, because there are two readers (§4) and a rule in only one of them would make the same file infer different types depending on its size.

Rendering to text. engine-core SpreadsheetRenderer is the only place the platform turns a spreadsheet into CSV, and it does so in process — never as a file, never as a service. Two things need it and both need the same bytes: a self-contained Power BI artifact carries its rows inside Csv.Document, and the Power BI model proves types and key uniqueness by scanning rows, which it can only do over text. Its contract is the round trip — parsing the rendered bytes as CSV yields the schema and rows the spreadsheet itself yields — and it always writes a comma, UTF-8 without a BOM, and \n.

That is also the answer to "why not convert everything to CSV up front": a CSV intermediate loses what the cell knows, and a second place that decides delimiter, quoting and encoding is the defect Delimiters already records.

4. Two Excel readers

Below generator.excel.streaming-threshold-bytes (16 MB) a workbook is built as a document. Above it, an .xlsx is walked as a stream of XML events by StreamingWorkbookReader: a document reader holds every row, cell and style at once, and the HTML target accepts 110 MB while Power BI accepts 510 MB.

The streaming reader does not use POI's XSSFSheetXMLHandler, which hands out one already-formatted string per cell. The rules above need the raw value and the format together, so the sheet XML is read directly and both readers end at the same string for the same cell. ExcelStreamingPathTest asserts exactly that.

An .xls past the same threshold is refused rather than streamed: BIFF8 caps a sheet at 65 536 rows, so one that large is unusual, and it is refused before POI opens it rather than after the heap is gone.

generator.excel.min-inflate-ratio (0.001) relaxes POI's zip-bomb guard, whose 1 % default is tuned for documents: a data sheet of repeated codes compresses far past it and was refused as an attack. The entry-count and total-size guards keep POI's defaults and are what actually bound the damage.

4a. Postal codes and other identifiers

A column of codes is read as text, never as a number, whatever its values look like in the first few hundred rows: PLZ, Postleitzahl, ZIP, Postcode, and the No./Nr/Code/SKU family. A postal code is an identifier that happens to be written with digits, and reading 01067 as the number 1067 loses the leading zero that Dresden needs — along with every join built on the column.

That is also what lets the platform recognise the column and look it up: a PLZ column becomes Bundesland, Ort and the state's ISO code before the dashboard is designed. See the reference tables page.

4b. An Apache Superset export

An export describes a dataset — its query, its columns, its metrics — and carries no rows. When the Dataset step is built, the platform writes a new dataset from it with eight date columns added for every date the query returns:

With more than one date column the labels name their source — Month (order_date), Day (shipped_at) — so each can be told apart in Apache Superset's column list.

Written in your database's SQL. The expressions follow the database the export names: PostgreSQL, MySQL and MariaDB, and Microsoft SQL Server each get their own date functions. Any other database is written as PostgreSQL, which Postgres-compatible warehouses understand.

What survives from your dataset. Your calculated columns keep their expressions, your columns keep their labels and descriptions, and the dataset keeps its schema, its time column, its cache timeout and your own metrics. A calculated date column gains no date parts, because the query has no such column to compute them from. When the Dataset step is skipped, your dataset is used exactly as it was uploaded.

4c. A Power BI model

A model you built in Power BI Desktop, handed over without its data, so the platform draws a report on your tables and measures instead of building a model of its own. Two ways to hand it over:

Left aside on read, by name: the .pbi folder (Desktop's local cache and settings — leave it out of the zip, it holds your rows and can push the upload past the size cap), an existing <Name>.Report, .git, any .pbix. A .pbix is not an upload at all: its model and its data share a part only Desktop can write, and the .pbit gives everything needed without the data.

What happens to it. The rows the agents design on are invented from the model — types, the date table, the relationships — and are in no file that comes back. The design agent references your measures by name and your columns as Table[Column]; it creates no measures of its own unless addMeasures is set, and those arrive as one file (definition/tables/_Measures.tmdl) beside the project rather than inside your folder. The download is a project: your model folder under its own name, a new report beside it whose definition.pbir points at that folder, and a README that ends with unzip, open the .pbip, Refresh — the visuals are empty until Desktop reads your sources. With reportOnly=true the model folder is left out, for a report placed beside the folder you already have; with modelConnection=<connection string> the report binds to a published model instead. A .pbit upload may also come back as a .pbit (outputFormat=PBIT): your model part byte-for-byte, with the new report layout.

Only Power BI. The HTML and Apache Superset targets refuse a model with "A Power BI model is generated for Power BI only — choose that target"; a model beside data files is refused too (the tables come from the model). The step-by-step draft builds the model itself and takes no model upload, and neither does the no-account trial.

5. What a refusal says

No library ever speaks to the user. DatasetParserFactory.guard turns a POI, Jackson or Parquet runtime failure into a sentence and logs the original at DEBUG — a job whose errorMessage read OLE2NotOfficeXmlFileException: The supplied data appears to be in the OLE2 Format… named a library, not a problem anybody could fix.

6. Adding a format

1. Add the constant to UploadFormat with its extensions and its trait.
2. Add a FileType constant and a @Component parser in parser implementing DatasetParser — parse for the schema, streamRows for every row, and sheets if it has any. Keep peak memory O(samples), not O(file); that is the contract CsvDatasetParser states and every parser inherits.
3. Map the two in DatasetParserFactory.toFileType and give it a phrase in describe.
4. Give MQueryFactory.SourceKind.of a case — a Power Query reader, or a deliberate refusal. An unrecognised name must never fall through to Csv.Document.
5. Nothing else. Studio's policies, the guest lists and the browser's accept follow the enum.

A format that carries no rows (the two Power BI model formats) is different: its parser refuses, and the target that reads it does so through its own reader (BundleExporter.readModel), before the parsers are asked. dataExtensions() leaves such a format out, so a data-only surface never offers it.

7. Deliberately not done

.xlsb. POI's XSSFBReader exists, but its sheet handler has the same problem as XSSFSheetXMLHandler — it yields formatted strings, not values with their formats — so honouring §3's rules would mean a hand-written BIFF12 record reader. POI cannot write an .xlsb either, so there is no way to build a fixture in the test suite and nothing to verify such a reader against. An .xlsb is therefore not in UploadFormat and is refused as an unsupported type, which is the honest answer until someone supplies a real file to test with.