> For the complete documentation index, see [llms.txt](https://docs.veilio.xyz/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.veilio.xyz/documentation/basics/datasets-uploads.md).

# Datasets Upload: CSV and JSON

Veilio datasets support both CSV and JSON upload formats.

Veilio datasets let you upload structured files, tokenize sensitive columns, and share filtered views with role-based access policies.

#### Supported formats

* **CSV** (`.csv`)
* **JSON** (`.json`) — array of objects

**Expected JSON shape**

```json
[
  { "email": "a@example.com", "phone": "+33600000000", "customer_id": "C001" },
  { "email": "b@example.com", "phone": "+33700000000", "customer_id": "C002" }
]
```

***

#### Upload flow

1. Owner or DPO uploads a file from the dashboard **or** via API.
2. Veilio extracts headers / columns from CSV or JSON.
3. `columnPolicies` must include **every** column exactly once.
4. Non-empty cell values are tokenized into the dataset grid.
5. A dataset version is created and available for controlled download.

**Identity column (`entityId`)**

Specify which column holds your business identifier (kept in clear text, links tokens to a profile):

* **Dashboard:** set during upload configuration.
* **API:** `entityIdColumn` in the multipart form (must match a column name).

**Historical migration upload**

For large legacy imports, pass `migration=true` so tokenized cells consume the **migration budget** instead of monthly plan quota.

Requirements:

* Migration mode activated (`POST /api/onboarding/migration`).
* Active Starter / Pro / Enterprise plan (or MIGRATION API key).

***

#### API upload (multipart)

**Endpoint:** `POST /api/datasets`

**Auth:** API key (`Authorization: Bearer` or `X-API-Key`) or dashboard session.

**Role:** Owner or DPO only.

**Content-Type:** `multipart/form-data`

| Form field       | Required | Description                                              |
| ---------------- | -------- | -------------------------------------------------------- |
| `file`           | Yes      | CSV or JSON file                                         |
| `name`           | Yes      | Dataset name (1–200 chars)                               |
| `columnPolicies` | Yes      | JSON **string** — one policy per column                  |
| `format`         | No       | `csv` or `json` (auto-detected from filename if omitted) |
| `description`    | No       | Max 2000 chars                                           |
| `entityIdColumn` | No       | Column name for business ID (plaintext)                  |
| `migration`      | No       | `"true"` to use migration budget                         |

**Example `columnPolicies`**

```json
{
  "customer_id": { "roles": ["OWNER", "DPO", "MEMBER", "VIEWER"] },
  "email": { "roles": ["OWNER", "DPO"] },
  "phone": { "roles": ["OWNER", "DPO", "MEMBER"] }
}
```

* `roles`: which organization roles may see this column on download.
* `userIds`: optional list of specific user IDs (alternative to roles).

Every column in the file **must** have a policy entry. Extra or missing keys return `VALIDATION_ERROR`.

**curl example**

```bash
export VEILIO_API_KEY="veil_live_xxxxxxxx"
export VEILIO_BASE_URL="https://app.veilio.fr/api"

curl -s -X POST "$VEILIO_BASE_URL/datasets" \
  -H "Authorization: Bearer $VEILIO_API_KEY" \
  -F "file=@contacts.csv" \
  -F "name=CRM Export March 2026" \
  -F 'description=Historical customer export' \
  -F 'entityIdColumn=customer_id' \
  -F 'migration=true' \
  -F 'columnPolicies={
    "customer_id": { "roles": ["OWNER", "DPO", "MEMBER", "VIEWER"] },
    "email": { "roles": ["OWNER", "DPO"] },
    "phone": { "roles": ["OWNER", "DPO", "MEMBER"] }
  }'
```

**JSON file example**

```bash
curl -s -X POST "$VEILIO_BASE_URL/datasets" \
  -H "Authorization: Bearer $VEILIO_API_KEY" \
  -F "file=@contacts.json;type=application/json" \
  -F "name=Contacts JSON" \
  -F 'format=json' \
  -F 'columnPolicies={"email":{"roles":["OWNER","DPO"]},"phone":{"roles":["OWNER","DPO","MEMBER"]}}'
```

**Success response `201`**

```json
{
  "id": "ds_...",
  "name": "CRM Export March 2026",
  "version": 1,
  "byteSize": 1048576,
  "columnCount": 3,
  "storage": "tokenized",
  "shareApiKey": "veil_...",
  "shareApiKeyPrefix": "veil_ab",
  "datasetAccessToken": "dsat_...",
  "datasetAccessTokenExpiresAt": "2026-03-28T10:00:00.000Z",
  "datasetAccessTokenHeader": "x-veilio-dataset-access-token"
}
```

`shareApiKey` and `datasetAccessToken` are returned only for **organization** datasets.

***

#### Validation rules and limits

| Rule             | Default / detail                                           |
| ---------------- | ---------------------------------------------------------- |
| Max file size    | **320 MB** (`VEILIO_DATASET_MAX_BYTES`)                    |
| Max cells        | **200 000** (`headers × rows`, `VEILIO_DATASET_MAX_CELLS`) |
| Empty file       | Rejected                                                   |
| `columnPolicies` | Must match columns exactly                                 |
| JSON shape       | Non-empty array of objects                                 |

#### Common errors

| Message                                    | Cause                           |
| ------------------------------------------ | ------------------------------- |
| `JSON dataset must be an array of objects` | Invalid JSON structure          |
| `JSON dataset is empty`                    | No rows                         |
| `Missing policy for column "..."`          | Incomplete `columnPolicies`     |
| `Policy references unknown column "..."`   | Extra policy key                |
| `Too many cells (max 200000)`              | File too large in cell count    |
| `File exceeds max size (... bytes)`        | File too large in bytes         |
| `Only Owner or DPO can upload datasets`    | Insufficient role               |
| `PLAN_LIMIT`                               | Dataset or token quota exceeded |
| Migration budget exhausted / expired       | When `migration=true`           |

***

#### Download behavior

Downloaded datasets are filtered by role and column policy. Consumers only receive columns they are allowed to see.

For programmatic download, use a Dataset Access Token (`dsat_...`) in header `x-veilio-dataset-access-token`.

**Related links**

* Dataset Access Tokens
* Historical Migration Mode
* API Reference