> For the complete documentation index, see [llms.txt](https://docs.veilio.xyz/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.veilio.xyz/documentation/basics/data-lifecycle.md).

# Data Lifecycle: Retention and Crypto Shredding

> **Important:** setting `retention` at tokenize time only **schedules** destruction. Tokens are shredded automatically **only if** a job calls `/api/internal/tokens/auto-shred` on a schedule. Without that job, expired tokens remain detokenizable.

***

#### Retention at tokenization time

When tokenizing data, you can define retention via:

| Field            | Type         | Description                                           |
| ---------------- | ------------ | ----------------------------------------------------- |
| `ttlDays`        | positive int | Shred after N days from tokenization                  |
| `retentionUntil` | ISO datetime | Absolute shred date (e.g. `2027-01-01T00:00:00.000Z`) |

Provide **one** of the two. Example:

```bash
curl -s -X POST "$VEILIO_BASE_URL/tokenize" \
  -H "Authorization: Bearer $VEILIO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "data": "john@example.com",
    "type": "email",
    "retention": { "ttlDays": 90 }
  }'
```

Response includes the resolved date:

```json
{
  "token": "tok_abc123...",
  "createdAt": "2026-07-26T08:00:00.000Z",
  "retentionUntil": "2026-10-24T08:00:00.000Z"
}
```

The same `retention` object works with `/tokenize/bulk`, `/tokenize/format`, and the JS/Python SDKs.

***

#### Auto-shredding (required for TTL / retentionUntil)

Internal route processes expired tokens in batches:

|          |                                               |
| -------- | --------------------------------------------- |
| Endpoint | `POST /api/internal/tokens/auto-shred`        |
| Auth     | `Authorization: Bearer $VEILIO_CRON_SECRET`   |
| Query    | `limit` (optional, default `500`, max `5000`) |

**What it does:** finds tokens where `retentionUntil <= now` and `shreddedAt` is null, then crypto-shreds them (encrypted payload cleared, detokenization returns `410 TOKEN_SHREDDED`).

**Configure the secret**

In your environment (required on **on-premise**; already set on Veilio SaaS):

```bash
VEILIO_CRON_SECRET=$(openssl rand -base64 32)
```

**Schedule the job**

Recommended: **hourly**. Example crontab on the host:

```cron
0 * * * * curl -fsS -X POST "https://your-domain.com/api/internal/tokens/auto-shred?limit=500" \
  -H "Authorization: Bearer $VEILIO_CRON_SECRET" \
  >> /var/log/veilio-auto-shred.log 2>&1
```

One-shot test:

```bash
curl -s -X POST "https://your-domain.com/api/internal/tokens/auto-shred?limit=500" \
  -H "Authorization: Bearer $VEILIO_CRON_SECRET"
```

Success response:

```json
{
  "processed": 12,
  "limit": 500,
  "executedAt": "2026-07-26T09:00:01.000Z"
}
```

If more than `limit` tokens expire at once, the next cron run continues the backlog.

| Environment                 | Who runs the cron?                                      |
| --------------------------- | ------------------------------------------------------- |
| **SaaS** (`app.veilio.xyz`) | Veilio — you only pass `retention` when tokenizing      |
| **On-premise**              | **You** — set `VEILIO_CRON_SECRET` and schedule the job |

See also: [Installation guide](https://docs.veilio.xyz/documentation/on-premise/installation-guide) (on-premise env + ops checklist).

***

#### On-demand shredding

Use `POST /api/tokens/shred` to shred a token immediately with an API key (GDPR erasure, early deletion, etc.).

```bash
curl -X POST "https://app.veilio.xyz/api/tokens/shred" \
  -H "Authorization: Bearer $VEILIO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"token": "tok_xxxxxxxxxxxx", "reason": "GDPR - user deletion request"}'
```

On-premise: replace the host with your instance URL (`https://your-domain.com/api/tokens/shred`).

> Do **not** use `DELETE /api/tokens/[id]` — that route is dashboard-only (browser session required) and will always return `401` when called with an API key.

**Request body**

| Field    | Type   | Required | Description                                       |
| -------- | ------ | -------- | ------------------------------------------------- |
| `token`  | string | Yes      | The `tok_...` value returned at tokenization time |
| `reason` | string | No       | Audit log label (max 500 chars)                   |

**Success (`200`)**

```json
{
  "token": "tok_xxxxxxxxxxxx",
  "shreddedAt": "2026-05-06T16:13:00.000Z"
}
```

**Error codes**

| HTTP | Code               | Cause                                       |
| ---- | ------------------ | ------------------------------------------- |
| 401  | `AUTH_ERROR`       | Missing, invalid, or revoked API key        |
| 404  | `NOT_FOUND`        | Token not found                             |
| 403  | `FORBIDDEN`        | Token belongs to a different organization   |
| 400  | `VALIDATION_ERROR` | Missing `token` field or invalid body shape |

SDK equivalents: `client.shredToken({ token, reason })` (JS) / `client.shred_token(token, reason=...)` (Python).

***

#### Detokenization behavior after shredding

Once shredded, detokenization returns:

* HTTP status: `410`
* code: `TOKEN_SHREDDED`

The operation is irreversible: encrypted data and search index are cleared.

***

#### Operational guidance

* Use retention defaults for sensitive classes (email / phone / ID).
* Keep shred `reason` values meaningful for auditability.
* On-premise: treat auto-shred as a **production go-live requirement** if you use `ttlDays` / `retentionUntil`.
* Monitor cron logs (`processed` count) so a silent auth failure (`401`) does not leave expired data live.
* Prefer hourly (or more frequent) runs for predictable compliance; increase `limit` if you ingest large batches with short TTLs.

***

#### Related links

* [SDK Javascript](https://docs.veilio.xyz/documentation/getting-started/sdk-js)
* [SDK Python](https://docs.veilio.xyz/documentation/getting-started/sdk-python)
* [HTTP Integration](https://docs.veilio.xyz/documentation/getting-started/http-integration)
* [Installation guide (on-premise)](https://docs.veilio.xyz/documentation/on-premise/installation-guide)
* [Go-live Checklist](https://docs.veilio.xyz/documentation/help/golive-check)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.veilio.xyz/documentation/basics/data-lifecycle.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.