Data architecture for AI: quality, RAG, pipelines and operations
From source to evaluation, the components required to supply an AI application with reliable, observable and governed data.
dataarchitectureragqualitymlops
Category
Data
Databases, pipelines, analytics, ML ops and data quality.
Databases, pipelines, analytics and MLOps are analysed as parts of one system. We cover reliability, governance and the quality required to support useful decisions.
Reference guide
Understand the subject
Latest coverage
Cloudflare saves 100 TB of memory by rebuilding the 1.1.1.1 DNS cache
Five Rust optimizations cut the footprint of a DNS entry by 56% at Cloudflare. Here are the lessons for systems operating at extreme scale.
CloudflareDNSRust
ClickHouse 26.7 adds EXPLAIN ANALYZE and faster Top-N queries
ClickHouse 26.7 exposes real query costs while optimizing Top-N aggregation and quantized vector search.
ClickHouseSQLEXPLAIN ANALYZE
Cloudflare Radar Researcher turns Internet data into a conversation
Cloudflare has opened the Radar Researcher beta, an assistant that queries network data in plain language and produces verifiable charts.
Cloudflare Radarartificial intelligenceInternet data
etcd 3.7.1: prepare an upgrade without losing quorum
etcd 3.7 adds RangeStream and performance improvements while removing legacy components. A safe upgrade procedure for Kubernetes operators.
etcdKubernetesmigration
PostgreSQL 19 Beta 2: what to test before the final release
PostgreSQL 19 Beta 2 sharpens the changes expected this fall. Performance, observability and compatibility: a practical, low-risk test plan.
PostgreSQLdatabasemigration
Doctolib and AI research: which health data, and how can users object?
Doctolib's health-data research project starts in August. Data scope, pseudonymisation and the right to object: what patients should check.
doctolibhealth dataprivacy
Database backups: RPO, RTO and restoration testing
PostgreSQL, MySQL, Docker and PITR: an operational method for choosing backups, testing restoration and proving RPO/RTO targets.
backuprestorepostgresql
RAG architecture: build, evaluate and operate it in production
Ingestion, hybrid search, reranking, citations and metrics: a complete method for building a reliable RAG system and deciding whether it is production-ready.
ragarchitectureevaluation
PostgreSQL 18: why uuidv7 and I/O matter to product teams
PostgreSQL 18 brings technical improvements that directly affect performance, migrations and data modeling.
postgresqldatabaseperformance
Apache Iceberg makes the lakehouse more credible
Iceberg brings snapshots, schema evolution and hidden partitioning to large analytical datasets.
lakehouseiceberganalytics
RAG and vector databases: quality starts with documents
Adding a vector database does not make an assistant reliable. Documents, chunks and metadata often determine the result.
ragvectorsai
Data quality: generative AI makes old problems more visible
Duplicates, incomplete fields and contradictory definitions become more dangerous when a model reformulates them confidently.
qualityaigovernance
Data observability: pipelines need traces too
Teams monitor APIs with traces and metrics. Data pipelines deserve the same rigor to understand delays and errors.
observabilitypipelinedata
Semantic layers stop revenue from being redefined in every dashboard
The semantic layer promises shared metrics across BI, notebooks and AI. It mostly requires clear governance.
analyticssemantic-layerbi
Real-time data: not every dataset needs streaming
Streaming is powerful, but it adds cost and complexity. The right pace depends on the decision being made.
streamingreal-timearchitecture
Data minimization is the forgotten reflex of AI projects
Teams want to give models more context. The best protection is sometimes not sending the data.
privacyaigovernance
Synthetic data is useful for testing and dangerous for reassurance
Synthetic data can protect privacy and speed up tests. It must still be compared with reality.
synthetic-datatestsprivacy
MLOps monitoring is more than server monitoring
Latency and availability are not enough. In production, a model can stay online while getting worse.
mlopsmonitoringai