[Lt-Flash]

Overview

This PR adds db_redis, a new optional OpenSIPS DB API backend that stores data in a Redis server or a Redis Cluster. Unlike cachedb_redis (which exposes Redis through the key-value NoSQL interface), db_redis presents Redis as a relational-style backend behind the standard db_url mechanism: a module is switched to Redis simply by pointing its db_url at a redis:// URL — no code change in the consuming module.

The module is excluded from the default build (it depends on hiredis) and is enabled by removing db_redis from exclude_modules, exactly like the other library-dependent DB drivers.

What the module covers

Data model — Redis is schemaless, so table layouts are declared out of band as Redis hashes:

• schema:<table> — ordered column list (__cols), primary key (__pk), and one type[,null][,auto] field per column (types: int, bigint, double, string, blob, datetime).
• <table>:<pk> — one row per key, one hash field per non-NULL column. Blobs are stored binary-safe (no encoding overhead); NULL is the absence of a field.
• seq:<table> — INCR counter backing auto-increment primary keys and last_inserted_id.
• The standard version table is provisioned like any other table.

Ready-to-load provisioning files ship under scripts/redis/.

Structured DB API implemented — query (primary-key fast path + full-table scan-and-filter, all six operators = < > <= >= != combined with AND/OR, single-column ORDER BY), insert (HSETNX row reservation for SQL-insert uniqueness), update (HSET/HDEL), delete (incl. bulk OR primary-key deletes and truncate), replace, insert_update, last_inserted_id. Typed results are produced from the declared schema.

Dual transport mode, auto-detected at startup via INFO CLUSTER:

• Single endpoint — a standalone server, or the current master reached through an external load balancer / Sentinel setup. Reconnects transparently on I/O errors and on -READONLY replies (a demoted master after failover).
• Redis Cluster — starting from a single seed host, the module discovers the full topology (CLUSTER SHARDS, falling back to CLUSTER SLOTS) and keeps a warm connection open to every master shard, so any key can be served immediately without a new TCP handshake on redirect. Keys are routed by CRC16 hash slot (honouring {hash tags}); MOVED is served from the warm pool with a deferred topology refresh, ASK is followed with a one-shot ASKING, and TRYAGAIN is retried after a short back-off. Full-table operations scan every master and merge. Only masters are used (read-your-writes consistency).

The mode can be pinned with the mode modparam (auto | single | cluster), turning a misconfiguration into a clear startup error instead of silently wrong behaviour.

Modules supported in this PR

Module	Table(s)	Status
dialog	dialog (v12)	Provisioning + verified end-to-end (table-version check, full load, shutdown flush) — including against a live 3-master Redis Cluster
dispatcher	dispatcher (v9)	Provisioning + verified end-to-end (destination load & activation)
usrloc	location (v1013)	Provisioning + code change (see below) + verified end-to-end (contact load in sql-only mode)
clusterer	clusterer (v4), clusterer_bridge (v1)	Provisioning + schema-validated

usrloc change

usrloc had a single raw SQL query (get_domain_db_ucontacts, used to fetch live contacts for NAT pinging) that excluded any backend without raw-query support. This PR adds a structured-query fallback: when the bound DB module lacks DB_CAP_RAW_QUERY, the query is built through the structured API (column projection + expires > now) and the contact_id modulo partitioning is applied client-side while iterating rows. SQL backends keep the exact same raw-query path, so there is no behaviour change for existing setups — and this also makes usrloc work with db_text. Consequently DB_CAP_RAW_QUERY is no longer a hard requirement in sql-only mode.

Roadmap

db_redis module

• Phase 1 (this PR) — structured DB API, dual single/cluster transport, schema provisioning.
• Phase 2 — secondary indexes for non-primary-key lookups (index sets, or RediSearch ON HASH where available), FETCH result cursors, per-node pipelining for bulk operations.
• Phase 3 — optional SQL-subset raw_query, a dbschema generation target to emit schema:<table> files from db/schema/*.xml, async operations.

Module coverage

Supported now: dialog, dispatcher, usrloc, clusterer.

Ready with only a provisioning file (structured DB API, no code change needed): the large majority of DB consumers, e.g. acc, alias_db, auth_db, domain, domainpolicy, permissions, group, speeddial, dialplan, load_balancer, mqueue, msilo, ratelimit, uac_registrant, userblacklist, tls_mgm, b2b_entities / b2b_logic / b2b_sca, presence / presence_xml / rls / pua, call_center, imc, tracer, trie, qrouting, rate_cacher, emergency, fraud_detection, cpl_c, xcap / xcap_client. These only need their schema hashes provisioned.

Needs a per-module structured fallback (raw SQL today, like usrloc did): drouting, carrierroute, auth_jwt, sipcapture.

Not applicable:

• sqlops — executes arbitrary user-supplied SQL, which cannot be translated to Redis operations.
• cachedb_* — these serve the NoSQL cache interface rather than the structured DB API, so they are consumers of cachedb_redis, not of this module. One exception: cachedb_perf (cachedb_perf: high-performance local memory cache built on modern kernel features #4118) also persists whole collections through the structured DB API (perf_save / perf_load, db_mode), so it is a genuine db_redis consumer — verified end to end against this module, with its cachedb_perf table (collection, pkey, pvalue, expires) provisioned like any other. Note Redis takes no transaction, so that snapshot is not atomic the way it is on an SQL backend.
• db_virtual, db_cachedb, cachedb_sql, sql_cacher — DB-layer plumbing, not end consumers.
• tm — keeps no database table.

Testing

• db_redis builds cleanly; module loads and binds via db_bind_api.
• Verified end-to-end against redis-server (standalone) and a live 3-master / 3-replica Redis 8 cluster: seeded with a single node, all masters are discovered and their connections warmed up; dialog rows load and flush across non-seed masters via the slot map.
• dispatcher and usrloc load their tables end-to-end through db_redis.
• All shipped schemas pass a column/type-field consistency check.

Notes

• New module modules/db_redis/ with DocBook documentation and generated README.
• New provisioning files under scripts/redis/.
• db_redis added to exclude_modules in Makefile.conf.template (optional, off by default).

[Lt-Flash]

Refactored the module to comply with OpenSIPS conventions after an internal review of the memory model and DB API usage:

• Value marshalling now goes through the DB layer helpers (db_str2int/db_str2bigint/db_str2double and db_int2str/db_bigint2str/db_double2str) instead of raw strtoll/strtod/snprintf, matching how db_sqlite and the other DB backends convert values — so type conversion and its error handling are consistent across drivers.

The rest of the model was already compliant and was verified during the review:

• All allocations use the OpenSIPS package memory API (pkg_malloc/pkg_free/pkg_realloc); no raw libc allocation. pkg is correct here since DB connections are per-worker (each process opens its own in child_init), exactly like db_mysql/db_sqlite.
• The connection struct header matches struct pool_con field-for-field, so the core connection pool (db_do_init/db_do_close) manages it correctly.
• Result values are handed off with VAL_FREE set so the core's db_free_row reclaims driver-allocated DB_STR/DB_BLOB buffers; schema-owned column names are (correctly) not freed.
• Uses the core DB entry points (db_do_init, db_do_close, db_use_table) and honors CON_OR_OPERATOR/CON_OR_RESET.

Remaining libc usage is limited to non-DB-boundary work (internal numeric comparison in filter/sort, and Redis-protocol parsing such as MOVED/ASK and SCAN cursors), where the DB helpers do not apply.

Regression-tested after the change: dialog load + shutdown flush over db_redis remains clean.

[github-actions]

Any updates here? No progress has been made in the last 30 days, marking as stale.

[Lt-Flash]

Waiting for review by mainteiner

[github-actions]

Any updates here? No progress has been made in the last 30 days, marking as stale.

[Lt-Flash]

Still waiting for review

[razvancrainea]

I am wondering what is the real benefit of using this module, instead of db_cachedb with cachedb_redis. If there are big differentces, isn't it a better idea to integrate them in the existing modules, rather than adding a new one?

Best regards,
Răzvan

[Lt-Flash]

Hi Răzvan,

Thank you for the review.

The separation was deliberate: I meant this as a plain db_api driver in the same role as db_mysql or db_sqlite - same front API, same relational row shape, with the storage simply living in Redis. Rows are native Redis hashes (HSET <table>:<pk>, one field per column) described by a per-table schema, so results come back as properly typed db_res_t, and the driver is cluster-aware for the DB workload (slot map discovered at startup, connections to all masters, MOVED/ASK followed). It has been exercised with dialog, dispatcher, usrloc and cachedb_perf.

On that specific combination: unless I am misreading master, it cannot run today - db_cachedb delegates to the backend's db_*_trans callbacks and only cachedb_mongodb implements them, so cachedb_redis would fail at the first query. I mention that only as the current state; your underlying question stands either way.

And I think it is the right question. The part I cannot argue with is duplication: a separate driver means the Redis connection, cluster topology and MOVED/ASK code exist twice in the tree.

Three shapes, and I am happy with any:

1. Standalone db_redis (this PR) - a db_api driver beside db_mysql/db_sqlite; cost is the duplicated Redis transport.
2. Fold it into cachedb_redis - implement the five db_*_trans callbacks there and close this PR; one Redis module, but the cache driver takes on typed schemas and column queries too.
3. Middle road - keep the mapping engine separate but let cachedb_redis own the transport, so the Redis connection code exists once.

You know the module layout far better than I do, so I will gladly follow whichever you prefer - including dropping this in favour of integration. Just say which and I will rework it.

Best regards,
Yury

[razvancrainea]

Hi, Yuri!

Thanks a lot for your work! If possible, I would go with option 2, as that brings more flexibility to whatever redis operations we have, whether they are DB based or Cache based. If that's not possible, or it introduces a huge performance penalty, let's find a middle road - perhaps db_cachedb can advertise a different way to use it, or the other way around, based on capabilities.

Best regards,
Răzvan

[Lt-Flash]

Hi Răzvan,

Thanks - that is a clear steer, and I am happy to reshape it into option 2. I will rework this PR in place rather than open a new one.

Two things I ran into while planning it, both in db_cachedb rather than in Redis, and I would value your view before I start:

1. Capabilities that cannot cross db_cachedb. db_cachedb_bind_api() exports a fixed set (use_table/init/close/query/free_result/insert/delete/update/raw_query) and the core derives the caps from those pointers, so a backend behind it can only ever be QUERY|INSERT|DELETE|UPDATE|RAW_QUERY. This module currently also advertises LAST_INSERTED_ID, REPLACE and INSERT_UPDATE, and there is no *_trans callback for any of them. Should db_cachedb grow them, or are you happy for those three to be dropped on the Redis path?

2. db_cachedb advertises RAW_QUERY but always fails it. db_cachedb_raw_query() returns -1 unconditionally, yet because the pointer is non-NULL the core sets DB_CAP_RAW_QUERY. A consumer therefore takes the raw path and fails at runtime - this already affects cachedb_mongodb. It also matters here: the usrloc fallback in this PR only engages when the driver lacks that capability. Not exporting raw_query (or exporting it per backend) would fix both, and it sounds close to the capability-based advertising you had in mind.

Otherwise the port looks straightforward: the row/schema mapping and query tiers move across as the five callbacks, and cachedb_redis takes over the connection and cluster layer, which already does topology and MOVED. The one thing I will measure before settling is the connection warm-up - this module opens connections to every master in child_init so a redirect is served on a warm socket, whereas cachedb_redis connects lazily.

Best regards,
Yury

[Lt-Flash]

I pushed d9e527f (db_redis: size the update argv on the heap, not the stack).

rdb_apply_update() declared its HSET argv/argvlen pair as variable-length arrays sized from _un, the number of SET columns, so its stack use grew with a count the function does not bound (32 bytes per column). The pair is now pkg_malloc()'d once per update in db_redis_update(), next to numbufs and sharing its error path and free, and reused for every matched row. That is how rdb_store_row() already allocates its own argv/argvlen.

It is not a crash anyone can trigger today: through sqlops, pv_print_buf_size limits an update to a few thousand columns, and every other dbf.update() caller in the tree passes a fixed column array. I changed it because a driver entry point should not size its stack from an unbounded caller-supplied count.

The module builds with -Werror on gcc 14, and no variable-length arrays are left in modules/db_redis.