ENH: Add the POMDPs.jl extension for DiscreteDP

[oyamad]

Close #398

This PR adds the package extension QuantEconPOMDPsExt connecting our DiscreteDP to the JuliaPOMDP ecosystem.

• POMDPs.jl is a package that provides a core interface for working with MDPs (Markov decision processes) and POMDPs (partially observable Markov decision processes). (DiscreteDP is a finite MDP.)
• POMDPs and POMDPTools are added as weak dependencies (weakdeps).
• The extension QuantEconPOMDPsExt is activated only with using POMDPs, POMDPTools.
• The change is purely additive: no existing behavior changes.

Features

Modeling interface

This allows us to use the POMDPs.jl interface as a simple interface for specifying a DiscreteDP model (see also QuantEcon/QuantEcon.py#228). As an example, consider the Aiyagari model as described in https://julia.quantecon.org/multi_agent_models/aiyagari.html (compare the corresponding script in #402):

using QuantEcon, POMDPs, POMDPTools

# Model specification as a subtype of POMDPs.MDP{S,A}
struct Household{TZ<:MarkovChain,TA<:AbstractVector,TU} <:
        POMDPs.MDP{Tuple{Float64,Float64},Float64}
    r::Float64
    w::Float64
    sigma::Float64
    beta::Float64
    z_chain::TZ
    a_vals::TA
    u::TU
end

function Household(; r = 0.01,
                   w = 1.0,
                   sigma = 1.0,
                   beta = 0.96,
                   z_chain = MarkovChain([0.9 0.1; 0.1 0.9], [0.1; 1.0]),
                   a_min = 1e-10,
                   a_max = 18.0,
                   a_size = 200,
                   a_vals = range(a_min, a_max, length = a_size),
                   u = sigma == 1 ? x -> log(x) :
                       x -> (x^(1 - sigma) - 1) / (1 - sigma))
    return Household(r, w, sigma, beta, z_chain, a_vals, u)
end

# The POMDPs.jl interface
POMDPs.states(am::Household) =
    Iterators.product(am.a_vals, am.z_chain.state_values)
POMDPs.actions(am::Household) = am.a_vals
POMDPs.actions(am::Household, (a, z)::Tuple) =
    (a_new for a_new in am.a_vals if am.w * z + (1 + am.r) * a - a_new > 0)
POMDPs.reward(am::Household, (a, z)::Tuple, a_new) =
    am.u(am.w * z + (1 + am.r) * a - a_new)
POMDPs.transition(am::Household, (a, z)::Tuple, a_new) =
    SparseCat([(a_new, z_new) for z_new in am.z_chain.state_values],
              am.z_chain.p[findfirst(==(z), am.z_chain.state_values), :])
POMDPs.discount(am::Household) = am.beta

am = Household(; a_max = 20.0, r = 0.03, w = 0.956)

# A native DiscreteDP can be constructed from a POMDPs.MDP model
am_ddp = DiscreteDP(am)
results = QuantEcon.solve(am_ddp, PFI)

# The rest is the same as in #402
a_vals = am.a_vals
z_vals = am.z_chain.state_values
pf = DDPPolicyFunction(results)               # (a, z) -> next period assets
a_star = [pf((a, z)) for a in a_vals, z in z_vals]

# To obtain the controlled MarkovChain
mc = markov_chain(results)  # or mc = results.mc
K = sum(stationary_distributions(mc)[1] .* first.(mc.state_values))

Joining the JuliaPOMDP ecosystem

This extension exposes our solution methods through a POMDPs.Solver (DiscreteDPSolver), so we can use tools from the JuliaPOMDP ecosystem to work with the model and the computed policy.

# Household defined as above
am = Household(; a_max = 20.0, r = 0.03, w = 0.956)

# Our `POMDPs.Solver`
solver = DiscreteDPSolver(PFI)     # solver isa POMDPs.Solver
policy = POMDPs.solve(solver, am)  # policy isa POMDPs.Policy

# queries
a_vals = am.a_vals
z_vals = am.z_chain.state_values
s = (a_vals[117], z_vals[2])            # (a, z) ≈ (11.658, 1.0)
action(policy, s)                       # sigma(a, z)
value(policy, s)                        # v(a, z)

a_star = [action(policy, (a, z)) for a in a_vals, z in z_vals]  # same table as above

# To obtain the controlled MarkovChain
mc = markov_chain(policy)
K = sum(stationary_distributions(mc)[1] .* first.(mc.state_values))

See "Interacting with Policies" in the POMDPs.jl documentation for details on working with the policy object, and "Implemented Simulators" for the tools that consume it (note simulate, like solve, must be qualified as POMDPs.simulate).
The native DPSolveResult remains reachable as policy.res.

Questions

1. Do we allow this package extension, with POMDPs and POMDPTools as weakdeps?
2. This PR will in effect have our DiscreteDP join the JuliaPOMDP ecosystem. Are we comfortable with that?

[oyamad]

Detailed description about the implementation, generated by Claude Code Fable 5:

---

Implementation notes

• The importer DiscreteDP(m::POMDPs.MDP; sparse=Val(true)) is index-free: it enumerates states(m) and actions(m) and builds its own value-to-index maps (stateindex/actionindex/initialstate are not consumed), attaching the enumerations as state_values/action_values from ENH: Add state_values/action_values to DiscreteDP #402 so that policies decode to model objects. The default output is the state-action pair form with a sparse transition matrix; sparse=Val(false) builds the dense product form directly.
• Rewards are the expected reward(m, s, a, sp) under transition(m, s, a) (3-argument models are covered by the POMDPs.jl fallback). Terminal states are encoded as zero-reward self-loops under every global action, so their value is exactly zero and actions(m, s) may be empty at terminal states. Transition weights are taken as-is (only exactly-zero branches are dropped), and numerical data are normalized to Float64, the convention of the ecosystem's own tabulation (SparseTabularMDP).
• The core gains only entry points: the markov_chain(res) accessor, the DiscreteDPSolver function stub whose MethodError hint directs users to load the trigger packages, the two exports (markov_chain, DiscreteDPSolver), and the Project.toml weakdeps entries. Everything else lives in the extension.
• DiscreteDPSolver(algo; sparse, max_iter, epsilon, k) carries the algorithm and the native solve options; the returned DiscreteDPPolicy is queried by state value, and the native DPSolveResult stays reachable as policy.res.
• Model export (as_mdp, exposing a DiscreteDP as a POMDPs.MDP) ships internal, as round-trip test infrastructure; its public naming is deferred.

Documentation and tests

• API page docs/src/api/QuantEconPOMDPsExt.md; worked example ext/examples/ifp_transient_shocks_pomdps.jl, the POMDPs.jl twin of examples/ifp_transient_shocks.jl from ENH: Add state_values/action_values to DiscreteDP #402, with the discretization (quadrature plus Young's lottery) inside transition — the model, not the tabulation, owns it. It is placed under ext/examples/ because it requires the weakdep trigger packages.
• test/test_pomdps.jl runs last in the suite (loading POMDPs makes the bare solve ambiguous in Main; test_ddp.jl tests the method-less stub and its hint before the extension loads): importer structure and expected values, terminal states with empty actions(m, s) in both storage forms, the sparse option, cross-validation against POMDPTools' SparseTabularMDP, validation errors (duplicate states, transitions leaving the state space), round trips through the internal as_mdp, and the solver/policy surface. POMDPs and POMDPTools are test-target dependencies, so regular CI exercises the extension; the solver also passes POMDPTools' test_solver smoke test on SimpleGridWorld and TabularMDP/RandomMDP.

For the release notes: new feature — the QuantEconPOMDPsExt extension, with new exports markov_chain and DiscreteDPSolver.

🤖 Generated with Claude Code (Claude Fable 5)

[Copilot]

Pull request overview

Adds an optional POMDPs.jl extension that converts finite MDP models to DiscreteDP and exposes QuantEcon solvers as POMDP policies.

Changes:

• Adds POMDP model tabulation, solver, policy, and controlled-chain integration.
• Registers POMDPs/POMDPTools as weak dependencies.
• Adds tests, API documentation, and an income-fluctuation example.

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
Project.toml	Configures extension dependencies and test targets.
ext/QuantEconPOMDPsExt.jl	Implements the POMDP bridge.
ext/examples/ifp_transient_shocks_pomdps.jl	Demonstrates extension usage.
src/QuantEcon.jl	Exports APIs and registers an error hint.
src/markov/ddp.jl	Adds solver stub and Markov-chain accessor.
test/test_pomdps.jl	Adds extension tests.
test/test_ddp.jl	Tests core APIs without the extension loaded.
test/runtests.jl	Runs POMDP tests last.
docs/Project.toml	Adds documentation dependencies.
docs/make.jl	Loads and documents the extension.
docs/src/api/QuantEconPOMDPsExt.md	Adds the extension API page.

Suppressed comments (1)

ext/QuantEconPOMDPsExt.jl:179

• This docstring is attached to a type, but it presents a constructor signature and omits the required # Fields section. Document the type signature and its three stored fields; constructor usage can remain in the prose or receive a separate method docstring.

    DiscreteDPSolver(algo=VFI; sparse=Val(true), max_iter=250,
                     epsilon=1e-3, k=20)

POMDPs.jl solver based on the `DiscreteDP` solution methods. `algo` is
one of `VFI`, `PFI`, or `MPFI`; `sparse` selects the tabulation
formulation (`Val(true)` for state-action pair form with sparse
storage, `Val(false)` for dense product form; value-typed, and carried
as a type parameter of the solver); the remaining keyword options are
those of `solve`. `POMDPs.solve(solver, m)` tabulates `m` via
`DiscreteDP(m)`, solves it, and returns a `DiscreteDPPolicy`.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.