We ship code and data everywhere, yet we still copy‑paste context in DMs. If your dataset loses its schema, if your model sheds its training sources, or if your prompt template gets detached from its intended metrics, your downstream users inherit guesswork. Open Knowledge Format (OKF) is a simple, pragmatic way to stop that loss. It packages knowledge artifacts—datasets, models, prompts, graphs, and their notes—together with standardized metadata, provenance, and signatures that travel intact.
Think of OKF as a lightweight wrapper and manifest, not a walled garden. It reuses proven web standards for identity and graphs and keeps the barrier to entry low: you can learn it in an afternoon, and you can adopt just the pieces that help you today.
What OKF Is—and Why Now
Open Knowledge Format is a spec and set of conventions for bundling an artifact plus the story of how and why it exists. AI has pushed this need into the mainstream. Agents generate flows that intermix data pulls, code execution, human edits, and model calls. Teams ship models trained on blended corpora. Cities publish portals with tables, dashboards, and reports that need durable context. Without a shared way to describe identity, lineage, license, and structure, we get data drift, duplicated effort, and weak trust.
OKF is intentionally boring by design. It uses JSON‑LD for linked metadata, SPDX for licenses, common checksum algorithms for integrity, and optional content credentials for signed provenance. You can keep your CSVs, Parquet, images, or model files exactly as they are. OKF adds a manifest and a few guardrails so your knowledge can move between people, tools, and time horizons without becoming ambiguous.
The OKF Core: Three Layers That Travel Together
OKF groups requirements and nice‑to‑haves into three layers. You can start with layer one and grow.
Layer 1: Identity and Packaging
This layer ensures every artifact has a stable handle and a portable envelope. Core elements include:
- Stable ID: a URI, DOI, or content identifier. Examples: https://example.org/datasets/okf-demo, a doi:10.1234/xyz, or a content address such as an IPLD CID.
- Manifest file: okf.jsonld at the root. Human‑readable, machine‑friendly, and versioned.
- Attachments: referenced files (CSV, Parquet, PNG, ONNX, safetensors, markdown, etc.) with byte sizes and checksums.
- Packaging: directory on disk, a TAR/ZIP, a CAR for content addressed storage, or a simple web root. No forced archive type.
Layer 2: Description and Graph
This layer provides meaning. It ensures that people and tools can answer: what is this, who made it, what is inside, and how does it relate to other things?
- JSON‑LD context to map keys to well‑known vocabularies (Schema.org, DataCite, Dublin Core) while staying concise.
- Creators with identifiers (ORCID if available), affiliations, and contact options that don’t leak private emails.
- License by SPDX identifier (e.g., CC-BY-4.0, MIT). If multiple assets carry different terms, list per‑file terms.
- Structure: links to a JSON Schema for tabular data or a SHACL shape for graphs; for models, a short card referencing inputs/outputs and on‑device limits.
- Relations: graph edges such as derivedFrom, cites, duplicates, supersedes, and isPartOf.
- Embeddings and indexes with vector dimensions, model names, and metric (e.g., cosine), not the vectors in‑manifest.
Layer 3: Integrity, Provenance, and Policy
This layer upgrades trust. It covers cryptographic checks, signatures, consent, and sensitive content markers.
- Checksums: per‑file SHA‑256 at minimum, optional BLAKE3 for speed.
- Signatures: a C2PA content credential or a standards‑based JSON Web Signature over the manifest for tamper evidence.
- Provenance events: machine‑readable steps such as collected, cleaned, fineTuned, with timestamps, tool versions, and agents involved.
- Policy hints: license rules, export warnings, or privacy flags signaled with ODRL‑style expressions and clear human text.
The Minimal OKF Profile You Can Adopt Today
If you implement only one thing, make it this small set in okf.jsonld:
- @id (URI or DOI), @type (Dataset, Software, Model, Document, Graph), name, description, version
- dateCreated, dateModified, inLanguage
- creator (name, ORCID if present), license (SPDX ID)
- attachments (path, mediaType, bytes, sha256)
That’s enough to be portable, indexable, and auditable. You can add lineage and signatures as you grow.
What Can Live in an OKF Package?
OKF is neutral about content types. The value comes from the manifest plus well‑known media types and checksums. Here are common patterns:
- Datasets: CSV/Parquet files, a JSON Schema, a data dictionary in markdown, and sample queries. Add derivedFrom links for source registries.
- Models: ONNX or safetensors, a lightweight model card, input/output schema, quantization details, and a test vector folder.
- Prompts and templates: prompt files with intended use, safety constraints, and evaluation metrics or datasets they presume.
- Knowledge graphs: Turtle or JSON‑LD graph files, a SHACL shape, and a set of named edges to external IRIs for joinability.
- Experiments: notebooks, script parameters, and run.json describing seeds, hardware notes, and reproducibility claims.
- Indexes: vector index shards or ANN structures with model references, embedding space dimensions, and recall tradeoffs noted.
OKF Aligns—It Doesn’t Replace
OKF is a glue layer. It embraces and references existing efforts rather than inventing its own universe.
Spatial and Remote Sensing: STAC
When packaging geospatial assets, reference STAC Items and Collections in the manifest. Let STAC handle geometry, time, and rasters. OKF carries your broader narrative: license nuances, derived tables, or model products that use the imagery.
Product and Supply Chains: GS1 Digital Link
When cataloging physical goods or digital twins, place the GS1 Digital Link in the manifest so downstream systems can resolve identity, safety sheets, or warranty PDFs.
People and Credentials: Verifiable Credentials
If you need to assert who signed or published the package, link a W3C Verifiable Credential that proves publisher identity without doxxing personal emails.
Security and IP: SPDX and SBOMs
List license by SPDX ID and attach an SBOM for software within the package. The manifest can point to SPDX documents that live alongside your binaries or models.
Media Integrity: Content Credentials
For image, audio, or video derivatives, use C2PA to embed a signed claim in the media itself. The OKF manifest includes the C2PA claim reference and additional narrative that C2PA doesn’t carry.
Make an OKF Package in a Day
You don’t need a committee to get started. Treat this as a practical checklist you can run in a small team or even solo.
1) Prepare your assets
- Decide what’s in scope: a dataset, a model, or a small bundle that tells a coherent story.
- Collect files into a clean directory. Keep file names stable and meaningful.
- Normalize encodings (UTF‑8) and add media types where you can.
2) Create the manifest
- Add okf.jsonld to the root. Include id, name, version, creators, license, dates, and a list of attachments with sizes and SHA‑256 digests.
- Reference a JSON Schema if you have tabular data; document nullability and units in a short data dictionary.
- For models, include supported platforms, max context length, memory footprint, and a pointer to test vectors.
3) Add provenance and relations
- Note source registries (derivedFrom with URIs or DOIs), transformation steps, and tool versions.
- Link to any works you cite. Include cites edges with DOIs or stable permalinks.
- Record any known usage constraints beyond the license (e.g., safety caveats).
4) Compute checksums
- For each attachment, compute SHA‑256 and add it to the manifest.
- Optionally, add a fast hash like BLAKE3 for integrity checks at scale.
5) Optional: sign it
- Create a JSON Web Signature over the manifest or apply a C2PA claim for media assets. Store the signature next to the manifest.
- Record who signed and the public key location. If you use a hardware token or cloud KMS, mention that in a short security note.
6) Publish and register
- Upload the folder to a web server, object storage, or a content‑addressed system (IPFS/IPLD). Keep a stable canonical URL.
- If appropriate, mint a DOI via DataCite and place it in the manifest as the primary id.
- Announce a change log. Future versions should carry supersedes relations for clean upgrades.
Validation You Can Script
OKF shines when you make validation automatic. Add a simple CI job that fails on broken checksums, bad SPDX strings, or missing required fields.
Schema checks
- Use a JSON Schema to validate your manifest’s required keys and types. Treat this as your local “OKF profile.”
- If you publish RDF or JSON‑LD graphs, use SHACL to ensure your relations are consistent.
Graph and link checks
- Resolve all external URIs at build time and warn on 404s. Store a last‑checked timestamp to catch future rot.
- De‑duplicate attachments by hash to avoid wasted storage and mismatched copies.
Security and privacy checks
- Lint licenses against a policy: block “no‑derivatives” licenses in corp training sets if your policy requires remixability.
- Scan for PII patterns in tabular data and add privacyFlag markers so downstream users do not accidentally misuse sensitive fields.
Privacy, Safety, and Consent in OKF
Metadata is not ethics, but it can reflect good practice. OKF encourages explicit signals:
- Provenance statements that mention consent collection methods for human data or why a dataset is publicly shareable.
- Usage warnings for models—e.g., “Not for medical diagnosis”—and a short rationale for known risks.
- Redaction notes that identify what was removed and why, so downstream users understand coverage gaps.
These hints are not enforcement. They are bright labels and machine‑readable flags that help operators and audits do the right thing.
OKF for AI Agents, RAG, and Personal Knowledge
Agents need compact, well‑labeled, and portable memory. OKF turns knowledge into capsules: a manifest, files, relations, and signatures that an agent can ingest, cite, and ship between devices or teammates.
Better citations with graphs
Because the manifest is JSON‑LD, your agent can traverse cites edges and print precise attributions. If you ship embeddings, record vector model names and dimensions, so your retrieval layer won’t misinterpret distances after upgrades.
Local‑first and offline
OKF favors directories and simple archives. Index OKF folders locally, add ETags or hashes to cache results, and sync only changed files. When you reconnect, your agent can verify hashes and reconcile updates with supersedes edges.
Safer sharing
When you send a model and its dataset capsule to a collaborator, the checksums and optional signature give them confidence they are testing the same thing you built. No more “works on my machine” mysteries.
OKF in Organizations: Small Wins First
Enterprises and public agencies do not need sweeping migrations to benefit. Start by packaging high‑value but fragile artifacts: the dataset everyone copies, the notebook that proves a KPI, or the prompt library running in production. Add a manifest with a license, owner, and checksums. That alone eliminates guesswork and speeds audits.
Next, establish one OKF profile per asset type you create often: a dataset profile, a model profile, and a report profile. Each profile is just a JSON Schema that your CI enforces. Over time, you can link those packages into a knowledge graph where updates ripple predictably through downstream apps.
Profiles in the Wild: Four Concrete Patterns
Research dataset profile
Requires a DOI, creators with ORCIDs, a JSON Schema for each table, and a data dictionary. Encourages provenance events for collection and cleaning. Optional: C2PA claims if imagery is included.
Model profile
Requires model type, input/output formats, a short card describing evaluation datasets, and at least one signed test vector. Encourages derivedFrom edges to training corpora with licenses.
City budget profile
Requires clear license, agency owner contact, data freshness, and alignment to an open data catalog. Encourages per‑table quality notes and change logs between fiscal years.
Museum collection profile
Requires item IDs, media types, copyright status, and creator attributions. Encourages links to external authority files and C2PA signatures on derivative images.
Common Pitfalls—and What to Do Instead
Anti‑pattern: giant monoliths
Don’t stuff everything into a single 40 GB archive. Do: split into packages by use case, and relate them via isPartOf and derivedFrom edges.
Anti‑pattern: vague licenses
Don’t write “open license” in free text. Do: use an SPDX identifier and state exceptions per attachment if needed.
Anti‑pattern: broken URLs and rot
Don’t rely on ephemeral links. Do: add DOIs for canonical records and content hashes for mirrors. Validate links in CI.
Anti‑pattern: signatures that quietly expire
Don’t sign once and forget. Do: record key rotation plans, publish public keys, and re‑sign new versions with supersedes edges.
Anti‑pattern: embedding vectors without context
Don’t ship a bare .npy. Do: document the embedding model, dimension, normalization, and distance metric in the manifest.
Versioning and Upgrades
OKF encourages semantic versioning for the package and clear relationships:
- version: “1.2.0” for non‑breaking field additions; “2.0.0” if you remove or change meanings.
- supersedes: which previous package to treat as replaced.
- isVariantOf: when two packages are peers with different constraints (e.g., a quantized model variant).
Keep a short CHANGELOG.md that mirrors manifest changes in simple language.
How OKF Fits Your Toolchain
OKF avoids dictating programming languages or frameworks. You can implement support in hours:
- CLI: small tool to create hashes, check media types, and generate a manifest skeleton.
- Indexers: scripts that read manifests, send summaries to a search index, and store graph edges in your favorite triple store.
- Viewers: a simple web page that renders okf.jsonld with collapsible sections and provenance timelines.
- Integrations: load OKF manifests in pipelines to attach consistent context to metrics and alerts.
Why Simplicity Wins
Many ambitious standards stumble because the on‑ramp is steeper than the payoff. OKF picks defaults that most people already know or can guess. JSON‑LD is just JSON with a thin layer of meaning. SPDX license IDs are short and familiar. Checksums are table stakes. Signatures remain optional but valuable. You get 80% of the benefit on day one by writing down identity, license, attachments, and hashes.
Roadmap: Profiles, Test Vectors, and Interop
Any healthy format needs public test vectors. A suggested community roadmap:
- Profile registry: curated JSON Schemas for common asset types with examples.
- Conformance suite: sample OKF packages that good tools can read, validate, and sign consistently.
- Bridges: mappers between OKF and STAC, SBOM, or catalog APIs, so organizations can adopt without rip‑and‑replace.
- Security best practices: recipes for key management, content credentials, and redaction markers.
These elements keep the spec small while the ecosystem grows around it.
FAQ in Brief
Is OKF a new file format?
No. It’s a format for meaning not bytes. Your files stay as they are. OKF adds a manifest and conventions for identity, graph, and trust.
Can I use YAML instead of JSON‑LD?
Yes, but you lose native linked data features unless you translate to JSON‑LD. If you must start with YAML for readability, ship an equivalent JSON‑LD manifest too.
Do I need to adopt all three layers?
No. Many teams start with Layer 1 and 2. Layer 3 is optional but recommended for sensitive or high‑stakes contexts.
Will OKF lock me into a single vendor?
OKF leans on open standards and generic web tech. You can host and validate with plain tools, from a local folder to cloud object storage.
Putting It All Together
Open Knowledge Format is not another heavy standard that asks you to rebuild your house. It is a front door and a label you can add to the house you already have. By describing what an artifact is, who made it, how it came to be, and how to verify it, you give collaborators, agents, and future you a cleaner way to reuse knowledge. In practice, that means fewer Slack threads trying to identify “the right CSV,” fewer bugs from mismatched model files, and faster reviews because the provenance is explicit. Most of all, it keeps the context glued to the content.
Summary:
- OKF is a simple manifest‑based approach that makes datasets, models, prompts, and graphs portable and trustworthy.
- Three layers: identity/packaging, description/graph, and integrity/provenance/policy.
- Adopt a minimal profile: id, type, version, creators, license, attachments with hashes.
- Aligns with existing standards like JSON‑LD, SPDX, STAC, C2PA, and Verifiable Credentials.
- Works well for AI agents and RAG as portable, verifiable memory capsules.
- Easy validation with JSON Schema, SHACL, and checksum checks in CI.
- Start small in organizations: package high‑value artifacts first, then grow profiles.
- Avoid anti‑patterns: giant monoliths, vague licenses, broken links, and unsignalled embeddings.
- Roadmap: profile registry, conformance suite, bridges, and security recipes.
