xan (the CSV magician) is an open-source command-line tool written in Rust designed to process, analyze, and visualize CSV files and adjacent data formats directly within the terminal. Originally created as a fork of BurntSushi's xsv, it was almost entirely rewritten by SciencesPo's médialab to support web data collection and social science research workflows.

The main features of this tool include:

• High Performance & Low Memory: Built in Rust to handle large files (in the Gigabytes range) with minimal memory usage. It utilizes a SIMD CSV parser (simd-csv) and supports multithreaded parallel execution (parallel / p) to speed up processing across CPU cores.

• Custom Expression Language: Features its own lightweight expression language optimized specifically for CSV data. It executes faster than dynamically-typed languages like Python or JavaScript when evaluating custom conditions for filtering (filter), creating new columns (map), transforming cells (transform), or aggregating data (agg, groupby).

• Terminal Visualizations: Allows users to render visual representations directly inside the terminal, including histograms (hist), scatter plots and line charts (plot), heatmaps (heatmap), and ASCII sparklines (spark).

• Extensive File & Compression Support: Processes .csv, .tsv, .psv, .scsv, .cdx (web archive index), .ndjson/.jsonl, .mtx, and bioinformatics formats (.vcf, .gtf, .sam, .bed). It natively reads compressed files (.gz with .gzi index support, and .zst) and can convert to/from JSON, Excel, and NumPy arrays.

• Specialized Domain Utilities: Extends traditional tabular data wrangling with built-in tools for lexicometry and text processing (tokenize, vocab), graph and network generation (network), matrix conversion (matrix), and web scraping (scrape).

• Comprehensive Command Suite: Provides a wide array of composable CLI commands to preview (view, flatten), search and slice (search, slice), deduplicate (dedup), group and summarize (groupby, stats, frequency), and merge datasets (join, cat, merge).

• Cross-Platform Package Availability: Can be installed via Cargo, Homebrew (macOS), Scoop (Windows), Pacman (Arch Linux), Nix, Conda, or pre-built binaries, and includes shell completions for terminal environments like Zsh.