Mutexes and Semaphores on Linux, C++, x86, ARM, and DPDK

Published 2026-07-26 · Markdown source

A practical implementation guide covering portable C++ synchronization, Linux futexes, Linux-kernel locking, architecture-specific atomic behavior, and DPDK data-plane synchronization.

Combined Markdown edition: July 26, 2026

Scope: The examples are intended for learning, prototyping, and architecture discussions. For production application code, prefer standard-library, POSIX, Linux-kernel, or DPDK primitives unless a custom implementation is genuinely required.

Version note: DPDK APIs and build options vary by release. Verify names and availability against the headers and documentation installed with your target DPDK version.

TOC mode: This edition uses only plain Markdown headings and links. No inline HTML anchors are required. The webpage renderer must generate IDs for headings.

Synchronization concepts
Portable Cpp implementation
Linux futex foundation
Futex based mutex
Futex based counting semaphore
Using the futex primitives
Memory ordering rationale
X86 and ARM instruction mapping
Production limitations of the educational futex code
Linux kernel implementation
DPDK synchronization model
Minimal DPDK spin lock example
Cpp wrappers for DPDK locks
DPDK polling counting semaphore
Using the DPDK primitives
Runnable DPDK EAL lcore example
Building and running the DPDK example
DPDK behavior on X86 and ARM
Choosing the appropriate primitive
Comparison matrix
References

---

Synchronization concepts

Mutex

A mutex protects a critical section so that only one execution context owns the lock at a time.

Important properties:

A mutex normally has ownership semantics.
The thread or lcore that locks it should unlock it.
It is appropriate for protecting shared data structures and invariants.
A sleeping mutex blocks or parks a waiter; a spin mutex consumes CPU while waiting.

Counting semaphore

A counting semaphore represents a number of permits. Acquiring consumes one permit; releasing returns one or more permits.

Important properties:

A semaphore generally does not have ownership semantics.
A different thread may release a permit.
It is appropriate for bounded concurrency, resource pools, and producer/consumer coordination.
A counting semaphore initialized to one can resemble a binary lock, but it is not a mutex because ownership and error semantics differ.

Sleeping versus polling synchronization

Behavior	Sleeping primitive	Polling primitive
Waiter action	Parks in the OS or scheduler	Repeatedly checks shared state
CPU consumption while waiting	Low	Potentially one full CPU
Typical use	General-purpose threads, control plane, long waits	Dedicated data-plane cores, very short waits
Examples	`std::mutex`, futex-backed locks, POSIX semaphores	`rte_spinlock_t`, ticket locks, custom DPDK polling semaphore

---

Portable Cpp implementation

For application code, use std::mutex and, with C++20 or later, std::counting_semaphore. The same source can be compiled for Linux/x86-64 and Linux/AArch64; the standard library and compiler select the platform implementation.


#include <mutex>
#include <semaphore>

std::mutex data_mutex;

// At most four operations may execute concurrently.
std::counting_semaphore<64> available_slots{4};

int shared_value = 0;

void worker()
{
    available_slots.acquire();

    {
        std::lock_guard<std::mutex> guard(data_mutex);
        ++shared_value;
    }

    available_slots.release();
}

Build on x86-64:


g++ -std=c++20 -O2 -pthread example.cpp -o example

Cross-compile for AArch64:


aarch64-linux-gnu-g++ -std=c++20 -O2 -pthread example.cpp -o example-arm64

Why this is normally the preferred implementation

The standard library supplies the correct architecture-specific atomic operations.
The implementation can use Linux futexes or another optimized runtime mechanism.
It handles details such as spurious wake-ups, scheduling, ABI compatibility, and library integration.
std::lock_guard provides exception-safe unlocking through RAII.

---

Linux futex foundation

The following is Linux user-space code, not Linux-kernel code.

A futex is based on a 32-bit userspace word. The uncontended path executes with ordinary atomic instructions. A thread invokes FUTEX_WAIT only when it may need to sleep, and an unlock or semaphore release invokes FUTEX_WAKE when sleepers may need to be resumed.

The futex wait operation compares the word with an expected value before blocking. If the value has already changed, the kernel returns without sleeping. This compare-and-block behavior is what prevents a wake-up from being lost between a userspace check and entry into the kernel.

Futex waits can return because of a wake operation, a value mismatch, a signal, or a spurious event. The userspace condition must therefore always be rechecked in a loop.

Shared futex support code


#include <atomic>
#include <cerrno>
#include <climits>
#include <cstdint>
#include <cstdlib>

#include <linux/futex.h>
#include <sys/syscall.h>
#include <unistd.h>

/*
 * A futex is always a 32-bit word on Linux, including 64-bit systems.
 *
 * atomic_ref gives C++ atomic semantics to the word while leaving the
 * underlying object suitable for the futex syscall.
 */
struct FutexWord {
    static_assert(
        std::atomic_ref<std::uint32_t>::is_always_lock_free,
        "This implementation requires lock-free 32-bit atomics");

    explicit constexpr FutexWord(std::uint32_t initial = 0) noexcept
        : value(initial)
    {
    }

    alignas(std::atomic_ref<std::uint32_t>::required_alignment)
        std::uint32_t value;

    [[nodiscard]]
    std::atomic_ref<std::uint32_t> atomic() noexcept
    {
        return std::atomic_ref<std::uint32_t>(value);
    }
};

namespace detail {

int futex_wait(FutexWord& word, std::uint32_t expected) noexcept
{
    return static_cast<int>(
        ::syscall(
            SYS_futex,
            &word.value,
            FUTEX_WAIT_PRIVATE,
            expected,
            nullptr,
            nullptr,
            0));
}

int futex_wake(FutexWord& word, int number_to_wake) noexcept
{
    return static_cast<int>(
        ::syscall(
            SYS_futex,
            &word.value,
            FUTEX_WAKE_PRIVATE,
            number_to_wake,
            nullptr,
            nullptr,
            0));
}

/*
 * Return when:
 *
 * - another thread wakes us;
 * - the futex value changed before we entered the kernel; or
 * - a spurious wake-up occurred.
 *
 * The caller must always recheck its userspace condition.
 */
void wait_while_equal(
    FutexWord& word,
    std::uint32_t expected) noexcept
{
    for (;;) {
        const int result = futex_wait(word, expected);

        if (result == 0) {
            return;
        }

        if (errno == EAGAIN) {
            // Value no longer equals expected.
            return;
        }

        if (errno == EINTR) {
            // Interrupted by a signal; retry the wait operation.
            continue;
        }

        // EFAULT, EINVAL, and similar errors indicate a programming error.
        std::abort();
    }
}

void wake(FutexWord& word, int number_to_wake) noexcept
{
    if (futex_wake(word, number_to_wake) < 0) {
        std::abort();
    }
}

} // namespace detail

FUTEX_WAIT_PRIVATE and FUTEX_WAKE_PRIVATE are suitable only when all participants are threads in the same process. For interprocess synchronization, place the futex word in shared memory and use the corresponding non-private futex operations.

---

Futex based mutex

The mutex uses this state machine:


0 = unlocked
1 = locked, with no known sleeping waiters
2 = locked, with possible sleeping waiters


class FutexMutex {
public:
    FutexMutex() = default;

    FutexMutex(const FutexMutex&) = delete;
    FutexMutex& operator=(const FutexMutex&) = delete;

    void lock() noexcept
    {
        auto state = state_.atomic();

        /*
         * Fast path:
         *
         * Try to change 0 -> 1 entirely in userspace.
         */
        std::uint32_t expected = 0;

        if (state.compare_exchange_strong(
                expected,
                1,
                std::memory_order_acquire,
                std::memory_order_relaxed)) {
            return;
        }

        /*
         * Slow path:
         *
         * Set state to 2, indicating that a sleeping waiter may exist.
         */
        for (;;) {
            if (expected != 2) {
                expected =
                    state.exchange(2, std::memory_order_acquire);
            }

            /*
             * exchange() returned 0, so this thread changed 0 -> 2
             * and acquired the mutex.
             */
            if (expected == 0) {
                return;
            }

            /*
             * Sleep only while the state still equals 2.
             *
             * If an unlock happened just before this syscall, the
             * kernel sees a value other than 2 and returns EAGAIN
             * rather than putting us to sleep.
             */
            detail::wait_while_equal(state_, 2);

            /*
             * Try to acquire the mutex while keeping it in the
             * contended state. Keeping state == 2 means that the
             * next unlock will wake another waiter.
             */
            expected = 0;

            if (state.compare_exchange_strong(
                    expected,
                    2,
                    std::memory_order_acquire,
                    std::memory_order_relaxed)) {
                return;
            }
        }
    }

    [[nodiscard]]
    bool try_lock() noexcept
    {
        auto state = state_.atomic();
        std::uint32_t expected = 0;

        return state.compare_exchange_strong(
            expected,
            1,
            std::memory_order_acquire,
            std::memory_order_relaxed);
    }

    void unlock() noexcept
    {
        auto state = state_.atomic();

        /*
         * Uncontended case:
         *
         *     1 -> 0
         *
         * No syscall is needed.
         *
         * Contended case:
         *
         *     2 -> 1 -> 0
         *
         * Then wake one possible waiter.
         */
        if (state.fetch_sub(1, std::memory_order_release) == 1) {
            return;
        }

        state.store(0, std::memory_order_release);
        detail::wake(state_, 1);
    }

private:
    FutexWord state_{0};
};

Mutex operation summary

Attempt an atomic 0 -> 1 transition.
If successful, enter the critical section without a syscall.
If the mutex is busy, mark it contended with state 2.
Call FUTEX_WAIT while state remains 2.
After waking, retry acquisition.
On unlock, use the fast userspace path when state was 1.
When state indicates contention, publish state 0 and wake one waiter.

---

Futex based counting semaphore

This version stores the number of available permits in the futex word.


class FutexSemaphore {
public:
    explicit FutexSemaphore(std::uint32_t initial_count) noexcept
        : count_(initial_count)
    {
    }

    FutexSemaphore(const FutexSemaphore&) = delete;
    FutexSemaphore& operator=(const FutexSemaphore&) = delete;

    void acquire() noexcept
    {
        auto count = count_.atomic();

        std::uint32_t current =
            count.load(std::memory_order_relaxed);

        for (;;) {
            /*
             * Try to consume one available permit.
             */
            while (current != 0) {
                if (count.compare_exchange_weak(
                        current,
                        current - 1,
                        std::memory_order_acquire,
                        std::memory_order_relaxed)) {
                    return;
                }

                /*
                 * On CAS failure, current is automatically replaced
                 * with the value that was actually observed.
                 */
            }

            /*
             * There are no permits. Sleep while count is still zero.
             */
            detail::wait_while_equal(count_, 0);

            /*
             * Recheck after every wake-up.
             */
            current = count.load(std::memory_order_relaxed);
        }
    }

    [[nodiscard]]
    bool try_acquire() noexcept
    {
        auto count = count_.atomic();

        std::uint32_t current =
            count.load(std::memory_order_relaxed);

        while (current != 0) {
            if (count.compare_exchange_weak(
                    current,
                    current - 1,
                    std::memory_order_acquire,
                    std::memory_order_relaxed)) {
                return true;
            }
        }

        return false;
    }

    /*
     * The caller must ensure that adding update does not overflow
     * the 32-bit permit count.
     */
    void release(std::uint32_t update = 1) noexcept
    {
        if (update == 0) {
            return;
        }

        if (update > static_cast<std::uint32_t>(INT_MAX)) {
            std::abort();
        }

        count_.atomic().fetch_add(
            update,
            std::memory_order_release);

        /*
         * Correctness-first implementation: wake at most one waiter
         * for each newly added permit.
         *
         * This deliberately makes a syscall on every release.
         * Production implementations generally track waiter state
         * to avoid a syscall when no thread is asleep.
         */
        detail::wake(count_, static_cast<int>(update));
    }

private:
    FutexWord count_;
};

Semaphore operation summary

Read the permit count.
If the count is nonzero, atomically decrement it.
If the count is zero, wait with FUTEX_WAIT while it remains zero.
Recheck the count after every return from the syscall.
A release atomically increments the count and wakes up to the number of newly available permits.

---

Using the futex primitives


FutexMutex mutex;
FutexSemaphore slots{4};

int shared_value = 0;

void operation()
{
    slots.acquire();

    mutex.lock();
    ++shared_value;
    mutex.unlock();

    slots.release();
}

A safer C++ interface can add a lock-guard-compatible wrapper or use std::lock_guard<FutexMutex> directly because FutexMutex supplies lock() and unlock():


#include <mutex>

FutexMutex mutex;
int shared_value = 0;

void guarded_operation()
{
    std::lock_guard<FutexMutex> guard(mutex);
    ++shared_value;
}

---

Memory ordering rationale

Mutex acquisition

Successful mutex acquisition uses std::memory_order_acquire.

Acquire semantics prevent reads and writes in the protected critical section from being reordered before the successful lock operation.

Mutex release

Unlock uses std::memory_order_release.

Release semantics publish writes performed in the critical section before another execution context successfully acquires the mutex.

Semaphore acquisition and release

A successful permit decrement uses acquire semantics.
Adding permits uses release semantics.
Relaxed loads and failed compare-and-exchange operations do not themselves establish synchronization; the successful acquire operation does.

Why x86 and ARM differ internally

The C++ memory model is architecture-independent, but the generated instructions differ:

x86 has a relatively strong hardware memory model, so a release store can normally be an ordinary aligned store.
ARM has a weaker memory model, so the compiler uses acquire/release instructions or explicit barriers where necessary.

The source-level memory orders must still be correct even when the x86 assembly appears to require fewer barriers.

---

X86 and ARM instruction mapping

The application source does not need separate x86 and ARM implementations. The compiler lowers C++ atomic operations to the appropriate target instructions.

Representative mapping

C++ atomic operation	x86-64	AArch64 without LSE	AArch64 with LSE
Acquire compare-and-exchange	`lock cmpxchg`	`ldaxr` / `stxr` retry loop	`casa`
Acquire exchange	`xchg`	`ldaxr` / `stxr` retry loop	`swpa`
Release store	Ordinary `mov` store	`stlr`	`stlr`
Release fetch-add	`lock xadd`	`ldxr` / `stlxr` retry loop	`ldaddl`

The exact output depends on compiler version, optimization level, architecture flags, surrounding code, and whether the target guarantees Large System Extensions (LSE).

x86-64 mutex fast path

A representative 0 -> 1 compare-and-exchange can look like this:


xor     eax, eax              ; expected = 0
mov     edx, 1                ; desired = 1
lock cmpxchg dword ptr [rdi], edx
sete    al                    ; return whether comparison succeeded

A release unlock can be an ordinary store:


mov     dword ptr [rdi], 0

The locked read-modify-write instruction provides the atomicity and ordering needed for acquisition. x86's memory model permits a plain aligned store for the release operation.

Baseline AArch64 compare-and-exchange

Without LSE, a compiler normally uses an exclusive load/store retry loop:


retry:
    ldaxr   w1, [x0]          // load-exclusive with acquire semantics
    cbnz    w1, failed

    mov     w2, #1
    stxr    w3, w2, [x0]      // store only if exclusive monitor is valid
    cbnz    w3, retry

    mov     w0, #1
    ret

failed:
    clrex
    mov     w0, #0
    ret

Release unlock:


stlr    wzr, [x0]             // store-release zero
ret

AArch64 with LSE

With ARMv8.1-A LSE, the lock operation can use a hardware compare-and-swap instruction:


mov     w1, #0
mov     w2, #1
casa    w1, w2, [x0]          // compare-and-swap, acquire semantics

The exact operand behavior and register allocation depend on compiler output, but the important distinction is that LSE supplies single-instruction atomic read-modify-write operations instead of a software retry loop built from exclusive accesses.

ARMv7

A 32-bit ARMv7 implementation generally uses LDREX and STREX retry loops together with appropriate DMB barriers because ARMv7 does not provide AArch64's LDAXR and STLR forms.

Conceptually:


retry:
    ldrex   r1, [r0]
    cmp     r1, #0
    bne     failed

    mov     r2, #1
    strex   r3, r2, [r0]
    cmp     r3, #0
    bne     retry

    dmb
    mov     r0, #1
    bx      lr

failed:
    clrex
    mov     r0, #0
    bx      lr

This is conceptual assembly; compiler-generated sequences and barrier placement depend on the operation and requested memory order.

---

Production limitations of the educational futex code

The futex implementations demonstrate the essential algorithms but omit production features:

No FIFO fairness.
No priority inheritance.
No robust-mutex or owner-death recovery.
No timed waits.
No cancellation handling.
No adaptive spinning before sleeping.
No mutex-owner tracking.
No recursive-lock detection.
No permit maximum in the futex semaphore.
The semaphore performs a wake syscall on every nonzero release.
FUTEX_WAIT_PRIVATE restricts the examples to one process.
No lifecycle protection against destruction while another thread is waiting.

FUTEX_WAKE does not guarantee FIFO selection of waiters. Neither the educational mutex nor the semaphore should be assumed to be fair.

For production user-space code, prefer one of the following:


std::mutex
std::counting_semaphore
pthread_mutex_t
sem_t

For special requirements, consider the appropriate established API:

Priority-inheritance pthread mutexes or PI futex operations.
Robust pthread mutexes for owner-death detection.
Process-shared pthread synchronization objects for interprocess use.
Condition variables for predicate-based sleeping waits.

---

Linux kernel implementation

Inside the Linux kernel, do not invoke futex syscalls. Use the kernel locking APIs.


#include <linux/mutex.h>
#include <linux/semaphore.h>

static DEFINE_MUTEX(data_lock);
static struct semaphore available_slots;

static int initialize_component(void)
{
    sema_init(&available_slots, 4);
    return 0;
}

static void do_work(void)
{
    down(&available_slots);

    mutex_lock(&data_lock);
    /* Access protected kernel data. */
    mutex_unlock(&data_lock);

    up(&available_slots);
}

Kernel mutex

A Linux-kernel mutex has owner semantics and is intended for mutual exclusion. Its implementation can combine:

An atomic fast path.
Optional optimistic spinning.
A sleeping wait queue under contention.

Kernel semaphore

A Linux-kernel semaphore is a counting primitive without mutex ownership semantics.

Context restriction

Kernel mutexes and semaphores are sleeping locks. Do not acquire them in interrupt context, with preemption constraints that prohibit sleeping, or while holding a spin lock that requires an atomic context.

In non-sleepable kernel contexts, the relevant primitive is normally a spin lock or another context-specific synchronization mechanism.

---

DPDK synchronization model

DPDK applications commonly dedicate logical cores to poll-mode processing. For very short data-plane critical sections, DPDK provides busy-wait synchronization primitives such as:

rte_spinlock_t
rte_ticketlock_t
rte_mcslock_t

A DPDK spin lock is not a sleeping Linux mutex. A waiting lcore continues polling until the lock becomes available.

DPDK and semaphores

The DPDK API is centered on data-plane polling and does not necessarily provide a general-purpose semaphore abstraction equivalent to std::counting_semaphore in every release. The implementation below is therefore an application-level polling counting semaphore built from DPDK atomic and wait/pause facilities.

Use such a polling semaphore only when:

The participating lcores are intentionally dedicated to polling.
Permit hold times are short and bounded.
Oversubscription is avoided.
Burning CPU while waiting is acceptable.

Use a sleeping OS primitive for control-plane threads, long waits, blocking I/O, or oversubscribed systems.

---

Minimal DPDK spin lock example


#include <rte_spinlock.h>

static rte_spinlock_t data_lock = RTE_SPINLOCK_INITIALIZER;
static unsigned int shared_value;

static void update_shared_value(void)
{
    rte_spinlock_lock(&data_lock);

    ++shared_value;

    rte_spinlock_unlock(&data_lock);
}

The uncontended lock operation is an architecture-appropriate atomic acquisition. Under contention, the waiter repeatedly observes the lock and executes a pause or wait hint. Unlock publishes the protected writes with release semantics.

---

Cpp wrappers for DPDK locks

The following wrappers satisfy the C++ BasicLockable interface and can be used with std::lock_guard.


// dpdk_sync.hpp
#pragma once

#include <cassert>
#include <cstdint>

#include <rte_common.h>
#include <rte_pause.h>
#include <rte_spinlock.h>
#include <rte_stdatomic.h>
#include <rte_ticketlock.h>

/*
 * Fast, non-FIFO DPDK spin mutex.
 *
 * Suitable for:
 *   - very short critical sections;
 *   - dedicated EAL lcores;
 *   - low or moderate contention.
 *
 * It does not put the calling thread to sleep.
 */
class alignas(RTE_CACHE_LINE_SIZE) DpdkSpinMutex final {
public:
    DpdkSpinMutex() noexcept
    {
        rte_spinlock_init(&lock_);
    }

    DpdkSpinMutex(const DpdkSpinMutex&) = delete;
    DpdkSpinMutex& operator=(const DpdkSpinMutex&) = delete;

    void lock() noexcept
    {
        rte_spinlock_lock(&lock_);
    }

    [[nodiscard]]
    bool try_lock() noexcept
    {
        return rte_spinlock_trylock(&lock_) != 0;
    }

    void unlock() noexcept
    {
        rte_spinlock_unlock(&lock_);
    }

private:
    rte_spinlock_t lock_;
};

/*
 * Fairer DPDK mutex.
 *
 * Waiting lcores receive tickets and acquire the lock in ticket order.
 * This can reduce starvation but may cost more under contention.
 */
class alignas(RTE_CACHE_LINE_SIZE) DpdkTicketMutex final {
public:
    DpdkTicketMutex() noexcept
    {
        rte_ticketlock_init(&lock_);
    }

    DpdkTicketMutex(const DpdkTicketMutex&) = delete;
    DpdkTicketMutex& operator=(const DpdkTicketMutex&) = delete;

    void lock() noexcept
    {
        rte_ticketlock_lock(&lock_);
    }

    [[nodiscard]]
    bool try_lock() noexcept
    {
        return rte_ticketlock_trylock(&lock_) != 0;
    }

    void unlock() noexcept
    {
        rte_ticketlock_unlock(&lock_);
    }

private:
    rte_ticketlock_t lock_;
};

Using a wrapper with `std::lock_guard`


#include <mutex>

DpdkSpinMutex mutex;
int shared_value = 0;

void update()
{
    std::lock_guard<DpdkSpinMutex> guard(mutex);
    ++shared_value;
}

Cache-line alignment

Aligning lock objects to RTE_CACHE_LINE_SIZE can reduce false sharing when adjacent data is frequently modified by different lcores. Alignment alone does not fix contention on the lock's own cache line; it prevents unrelated objects from sharing that line.

Optional MCS lock for heavy contention

An MCS lock gives each waiter its own node and makes it spin on local state, reducing shared-cache-line bouncing. The node must remain alive for the complete lock hold, and each concurrent acquirer needs a distinct node.


#include <rte_mcslock.h>

static rte_mcslock_t* global_lock = nullptr;

void critical_section()
{
    // Each thread or lcore must use its own node.
    rte_mcslock_t local_node{};

    rte_mcslock_lock(&global_lock, &local_node);

    /* Very short protected work. */

    rte_mcslock_unlock(&global_lock, &local_node);
}

---

DPDK polling counting semaphore

This semaphore stores the number of available permits in a DPDK atomic variable.


// Add this to dpdk_sync.hpp

class alignas(RTE_CACHE_LINE_SIZE) DpdkCountingSemaphore final {
public:
    /*
     * Preconditions:
     *
     *   maximum > 0
     *   initial <= maximum
     */
    DpdkCountingSemaphore(
        std::uint32_t initial,
        std::uint32_t maximum) noexcept
        : maximum_(maximum)
    {
        assert(maximum_ > 0);
        assert(initial <= maximum_);

        rte_atomic_store_explicit(
            &count_,
            initial,
            rte_memory_order_relaxed);
    }

    DpdkCountingSemaphore(const DpdkCountingSemaphore&) = delete;
    DpdkCountingSemaphore&
    operator=(const DpdkCountingSemaphore&) = delete;

    /*
     * Consume one permit.
     *
     * This is a polling operation. It does not invoke futex(), sleep(),
     * or any other Linux blocking syscall.
     */
    void acquire() noexcept
    {
        for (;;) {
            if (try_acquire()) {
                return;
            }

            /*
             * Wait until count != 0.
             *
             * The mask covers the entire 32-bit value. The wait uses a
             * relaxed load because the successful CAS below performs
             * the actual acquire operation.
             */
            RTE_WAIT_UNTIL_MASKED(
                &count_,
                UINT32_MAX,
                !=,
                0u,
                rte_memory_order_relaxed);
        }
    }

    /*
     * Try to consume one permit without waiting.
     */
    [[nodiscard]]
    bool try_acquire() noexcept
    {
        std::uint32_t observed =
            rte_atomic_load_explicit(
                &count_,
                rte_memory_order_relaxed);

        while (observed != 0) {
            if (rte_atomic_compare_exchange_weak_explicit(
                    &count_,
                    &observed,
                    observed - 1,
                    rte_memory_order_acquire,
                    rte_memory_order_relaxed)) {
                return true;
            }

            /*
             * On CAS failure, observed is updated with the value that
             * is currently stored in count_.
             */
        }

        return false;
    }

    /*
     * Return one or more permits.
     *
     * Returns false if the update would exceed maximum_. Such a failure
     * normally means the caller released more permits than it acquired.
     */
    [[nodiscard]]
    bool release(std::uint32_t update = 1) noexcept
    {
        if (update == 0) {
            return true;
        }

        if (update > maximum_) {
            return false;
        }

        std::uint32_t observed =
            rte_atomic_load_explicit(
                &count_,
                rte_memory_order_relaxed);

        for (;;) {
            if (observed > maximum_ - update) {
                return false;
            }

            if (rte_atomic_compare_exchange_weak_explicit(
                    &count_,
                    &observed,
                    observed + update,
                    rte_memory_order_release,
                    rte_memory_order_relaxed)) {
                return true;
            }
        }
    }

    /*
     * Diagnostic snapshot only. The result can become stale immediately.
     */
    [[nodiscard]]
    std::uint32_t available() const noexcept
    {
        return rte_atomic_load_explicit(
            &count_,
            rte_memory_order_relaxed);
    }

    [[nodiscard]]
    std::uint32_t maximum() const noexcept
    {
        return maximum_;
    }

private:
    RTE_ATOMIC(std::uint32_t) count_;
    const std::uint32_t maximum_;
};

Why the loop is required

Multiple lcores can observe the same nonzero count. Only one can successfully perform a particular decrement. The others receive an updated observed value from the failed compare-and-exchange and retry.

Why the wait is still polling

RTE_WAIT_UNTIL_MASKED repeatedly tests an atomic condition while invoking the target architecture's pause or wait mechanism. It does not turn this abstraction into a Linux sleeping semaphore.

Fairness

The semaphore does not provide FIFO fairness. A lcore that begins retrying at the right moment can acquire a newly released permit before an older waiter.

---

Using the DPDK primitives


#include "dpdk_sync.hpp"

#include <cstdlib>
#include <mutex>

DpdkSpinMutex counter_mutex;

// Allow at most four lcores into the bounded operation.
DpdkCountingSemaphore operation_slots{4, 4};

std::uint64_t shared_counter = 0;

void process_operation()
{
    operation_slots.acquire();

    /*
     * Only four lcores can be in this overall operation concurrently.
     * The mutex separately protects the shared counter.
     */
    {
        std::lock_guard<DpdkSpinMutex> guard(counter_mutex);
        ++shared_counter;
    }

    if (!operation_slots.release()) {
        /*
         * A false return indicates an unbalanced release or another
         * programming error.
         */
        std::abort();
    }
}

RAII permit guard

A permit guard makes release exception-safe and reduces the chance of an unbalanced acquire/release pair.


class DpdkSemaphorePermit final {
public:
    explicit DpdkSemaphorePermit(DpdkCountingSemaphore& semaphore) noexcept
        : semaphore_(&semaphore)
    {
        semaphore_->acquire();
    }

    DpdkSemaphorePermit(const DpdkSemaphorePermit&) = delete;
    DpdkSemaphorePermit& operator=(const DpdkSemaphorePermit&) = delete;

    ~DpdkSemaphorePermit()
    {
        if (semaphore_ != nullptr) {
            const bool released = semaphore_->release();
            if (!released) {
                std::abort();
            }
        }
    }

    void release_early() noexcept
    {
        if (semaphore_ == nullptr) {
            return;
        }

        const bool released = semaphore_->release();
        if (!released) {
            std::abort();
        }

        semaphore_ = nullptr;
    }

private:
    DpdkCountingSemaphore* semaphore_;
};

Usage:


void process_with_raii()
{
    DpdkSemaphorePermit permit(operation_slots);

    std::lock_guard<DpdkSpinMutex> guard(counter_mutex);
    ++shared_counter;
}

---

Runnable DPDK EAL lcore example


// dpdk_sync.cpp

#include "dpdk_sync.hpp"

#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <mutex>

#include <rte_debug.h>
#include <rte_eal.h>
#include <rte_launch.h>
#include <rte_lcore.h>

namespace {

constexpr std::uint32_t iterations_per_lcore = 100'000;

/*
 * Only two lcores may hold a semaphore permit concurrently.
 */
DpdkCountingSemaphore slots{2, 2};

/*
 * The counter itself remains protected by a mutex.
 */
DpdkSpinMutex counter_mutex;
std::uint64_t counter = 0;

int worker_main(void*)
{
    for (std::uint32_t i = 0; i < iterations_per_lcore; ++i) {
        slots.acquire();

        {
            std::lock_guard<DpdkSpinMutex> guard(counter_mutex);
            ++counter;
        }

        if (!slots.release()) {
            std::abort();
        }
    }

    return 0;
}

} // namespace

int main(int argc, char** argv)
{
    const int eal_result = rte_eal_init(argc, argv);

    if (eal_result < 0) {
        rte_exit(EXIT_FAILURE, "Failed to initialize DPDK EAL\n");
    }

    unsigned int lcore_id;

    /*
     * rte_eal_remote_launch() is called from the main lcore and launches
     * worker_main() on every enabled worker lcore.
     */
    RTE_LCORE_FOREACH_WORKER(lcore_id) {
        const int result =
            rte_eal_remote_launch(worker_main, nullptr, lcore_id);

        if (result != 0) {
            rte_exit(
                EXIT_FAILURE,
                "Failed to launch worker on lcore %u\n",
                lcore_id);
        }
    }

    /*
     * Include the main lcore in the test.
     */
    worker_main(nullptr);

    RTE_LCORE_FOREACH_WORKER(lcore_id) {
        const int worker_result = rte_eal_wait_lcore(lcore_id);

        if (worker_result < 0) {
            rte_exit(
                EXIT_FAILURE,
                "Worker on lcore %u failed\n",
                lcore_id);
        }
    }

    const std::uint64_t expected =
        static_cast<std::uint64_t>(rte_lcore_count()) *
        iterations_per_lcore;

    std::printf(
        "counter:  %llu\n"
        "expected: %llu\n",
        static_cast<unsigned long long>(counter),
        static_cast<unsigned long long>(expected));

    const int cleanup_result = rte_eal_cleanup();

    return cleanup_result == 0 ? EXIT_SUCCESS : EXIT_FAILURE;
}

This example:

Initializes the DPDK Environment Abstraction Layer.
Launches a worker function on every enabled worker lcore.
Runs the same function on the main lcore.
Limits concurrent entry through the semaphore.
Protects the shared counter with a DPDK spin mutex.
Waits for all worker lcores.
Verifies the final count.

---

Building and running the DPDK example

DPDK installations normally expose compiler and linker flags through libdpdk.pc.


g++ -O3 -std=gnu++17 \
    $(pkg-config --cflags libdpdk) \
    dpdk_sync.cpp \
    -o dpdk_sync \
    $(pkg-config --libs libdpdk)

GNU C++ mode may be needed because some DPDK headers use GNU extensions.

A CPU-only test that does not access a NIC can commonly be launched as:


./dpdk_sync -l 0-3 --no-pci --no-huge

The exact EAL options depend on the DPDK version, platform, privileges, memory configuration, and application requirements.

Cross-compilation

The application source is architecture-neutral. For an AArch64 target:

Use an AArch64 cross-compiler or native AArch64 build environment.
Build or install DPDK for the same target and ABI.
Ensure pkg-config resolves the target DPDK metadata rather than host libraries.
Link the application against the target DPDK build.

---

DPDK behavior on X86 and ARM

x86 pause behavior

On x86, DPDK's pause helper maps to the processor pause hint, commonly _mm_pause() in C/C++ and pause in assembly.

Conceptually:


retry:
    lock cmpxchg [lock], desired
    je   acquired

wait:
    pause
    cmp  dword ptr [lock], 0
    jne  wait
    jmp  retry

The pause hint can improve behavior of tight spin loops by reducing pipeline penalties and resource pressure. It does not put the thread to sleep.

AArch64 pause behavior

A normal AArch64 pause helper can emit yield:


retry:
    ldaxr   w1, [x0]
    cbnz    w1, wait

    mov     w2, #1
    stxr    w3, w2, [x0]
    cbnz    w3, retry
    ret

wait:
    yield
    b       retry

Some DPDK/ARM configurations can use event-based waiting facilities such as WFE in appropriate wait helpers:


wait:
    wfe
    b       retry

WFE can reduce power consumption relative to an aggressive load loop, but the exact behavior depends on platform support, DPDK configuration, exclusive-monitor behavior, and the wait helper being used.

ARM LSE

On systems where ARM Large System Extensions are enabled for compilation, atomic operations can use instructions such as CAS, SWP, and LDADD. Without LSE, they are generally implemented using exclusive load/store loops.

Application-level implication

No architecture-specific lock code is normally required in the DPDK application. DPDK headers and compiler atomics select the appropriate:

Atomic read-modify-write instructions.
Memory barriers or acquire/release instruction forms.
Pause, yield, or optional event-wait behavior.

---

Choosing the appropriate primitive

Use `std::mutex` when

The code is ordinary C++ application or control-plane code.
Wait times may be nontrivial.
Threads can be descheduled.
Portability and maintainability are priorities.

Use `std::counting_semaphore` when

The program needs bounded concurrency or a permit pool.
C++20 is available.
A standard sleeping/waking implementation is appropriate.

Use a futex-based custom primitive when

You are studying low-level synchronization.
You need a specialized userspace ABI or state machine.
You can justify and test the substantial correctness burden.

For production, a mature library implementation is usually safer.

Use a Linux-kernel mutex or semaphore when

The code runs inside the Linux kernel.
Sleeping is permitted in the current context.
Mutex ownership or semaphore counting semantics match the requirement.

Use `rte_spinlock_t` when

The critical section is extremely short.
The holder cannot block or be descheduled for a long time.
Participating lcores are dedicated and not oversubscribed.
Minimal lock overhead matters more than fairness.

Use `rte_ticketlock_t` when

FIFO-style acquisition ordering is useful.
Reduced starvation is more important than the smallest lock footprint.
The contention level remains suitable for a single shared ticket state.

Use `rte_mcslock_t` when

Many lcores can contend for the same lock.
Cache-line bouncing on a centralized lock becomes expensive.
Per-waiter queue nodes can be managed correctly.

MCS waiters spin primarily on local state, which can scale better under heavy contention.

Use the custom DPDK polling semaphore when

The wait is intentionally polling.
Permit hold times are short and predictable.
Dedicated lcores are available.
Non-FIFO semantics are acceptable.

Do not use a polling primitive when

The lock holder may perform disk, network, file, or blocking system calls.
The holder may sleep or be preempted for long periods.
The process is oversubscribed.
A waiter may remain blocked for milliseconds or longer.
Energy consumption is important and event-based waiting is unavailable.

---

Comparison matrix

Primitive	Wait behavior	Ownership	Fairness	Cross-process	Best fit
`std::mutex`	Usually sleeping/adaptive	Yes	Implementation-defined	No, by itself	General C++ mutual exclusion
`std::counting_semaphore`	Usually sleeping/adaptive	No	Implementation-defined	No, by itself	General C++ bounded concurrency
Educational futex mutex	Sleeping under contention	Intended yes, not tracked	No guarantee	Private version: no	Learning/custom Linux userspace
Educational futex semaphore	Sleeping under contention	No	No guarantee	Private version: no	Learning/custom Linux userspace
Kernel `struct mutex`	Kernel sleeping lock	Yes	Kernel implementation policy	Kernel internal	Kernel mutual exclusion
Kernel `struct semaphore`	Kernel sleeping lock	No	Kernel implementation policy	Kernel internal	Kernel permit counting
`rte_spinlock_t`	Busy wait	By convention	Not FIFO	Shared-memory dependent	Very short DPDK critical sections
`rte_ticketlock_t`	Busy wait	By convention	Ticket order	Shared-memory dependent	Fairer DPDK locking
`rte_mcslock_t`	Busy wait on queue node	By convention	Queue-based	Shared-memory dependent	Highly contended DPDK locking
Custom DPDK semaphore	Busy wait	No	No guarantee	Shared-memory dependent	Short bounded data-plane operations

Decision rule

A useful default decision sequence is:

Use a lock-free or ownership-partitioned design when it simplifies the data plane.
Otherwise use the highest-level standard or platform primitive that satisfies the requirement.
Use a spin-based DPDK primitive only for short, bounded, data-plane waits.
Use custom atomic/futex implementations only after defining invariants, lifecycle rules, fairness expectations, failure behavior, and a serious stress-testing strategy.

---

References

C++ synchronization and memory model

[C++ draft: mutex requirements](https://eel.is/c++draft/thread.mutex.requirements)
[C++ draft: multithreading and memory model](https://eel.is/c++draft/intro.multithread)

Linux futexes

[Linux manual: futex(2)](https://man7.org/linux/man-pages/man2/futex.2.html)
[Linux manual: FUTEX_WAKE](https://man7.org/linux/man-pages/man2/FUTEX_WAKE.2const.html)

Linux-kernel locking

[Linux kernel documentation: mutex design](https://docs.kernel.org/locking/mutex-design.html)

ARM synchronization

[Arm instruction reference: LDAXR](https://developer.arm.com/documentation/ddi0596/latest/Base-Instructions/LDAXR--Load-Acquire-Exclusive-Register-)
[Arm synchronization primitives: LDREX and STREX](https://developer.arm.com/documentation/dht0008/latest/arm-synchronization-primitives/exclusive-accesses/ldrex-and-strex)

DPDK APIs and guides

[DPDK API: spin locks](https://doc.dpdk.org/api/rte__spinlock_8h.html)
[DPDK API: ticket locks](https://doc.dpdk.org/api/rte__ticketlock_8h.html)
[DPDK API: MCS locks](https://doc.dpdk.org/api/rte__mcslock_8h.html)
[DPDK API: launch functions](https://doc.dpdk.org/api/rte__launch_8h.html)
[DPDK Programmer's Guide: thread safety](https://doc.dpdk.org/guides/prog_guide/thread_safety.html)
[DPDK Linux Getting Started Guide: building applications](https://doc.dpdk.org/guides/linux_gsg/build_dpdk.html)
[DPDK API index](https://doc.dpdk.org/api/)

---

Closing guidance

For ordinary Linux C++ software, start with std::mutex and std::counting_semaphore. For Linux-kernel code, use the kernel locking primitives appropriate to the execution context. For DPDK data-plane code, first try to avoid shared writable state through per-lcore ownership, queues, rings, or batching; when a shared lock is unavoidable, keep the protected section short and select the lock according to contention and fairness requirements.