|
| 1 | +diff --git a/.lycheeignore b/.lycheeignore |
| 2 | +index 3c81b5b..aaae477 100644 |
| 3 | +--- a/.lycheeignore |
| 4 | ++++ b/.lycheeignore |
| 5 | +@@ -22,3 +22,13 @@ https://www.uber.com/blog/deep-neuroevolution/ |
| 6 | + # Ignore localhost links Serving tutorial |
| 7 | + http://localhost:8000 |
| 8 | + http://127.0.0.1:8000 |
| 9 | ++ |
| 10 | ++# pytorch.org/cppdocs pages return 404 to the lychee-action CI runner (likely |
| 11 | ++# bot/rate-limit blocking on pytorch.org's CDN) but resolve fine (HTTP 200) |
| 12 | ++# for a normal client. Verified manually before adding here — see issue #3911. |
| 13 | ++https://pytorch\.org/cppdocs/.* |
| 14 | ++ |
| 15 | ++# NNCF autoapi anchors flagged in issue #3911: the base page and both anchors |
| 16 | ++# (nncf.experimental.torch.fx.OpenVINOQuantizer, .quantize_pt2e) exist and |
| 17 | ++# render correctly; likely blocked/rate-limited for the CI runner same as above. |
| 18 | ++https://openvinotoolkit\.github\.io/nncf/autoapi/nncf/experimental/torch/fx/index\.html.* |
| 19 | +diff --git a/intermediate_source/dist_tuto.rst b/intermediate_source/dist_tuto.rst |
| 20 | +index cebc986..0bcd9da 100644 |
| 21 | +--- a/intermediate_source/dist_tuto.rst |
| 22 | ++++ b/intermediate_source/dist_tuto.rst |
| 23 | +@@ -37,7 +37,7 @@ In order to get started we need the ability to run multiple processes |
| 24 | + simultaneously. If you have access to compute cluster you should check |
| 25 | + with your local sysadmin or use your favorite coordination tool (e.g., |
| 26 | + `pdsh <https://linux.die.net/man/1/pdsh>`__, |
| 27 | +-`clustershell <https://cea-hpc.github.io/clustershell/>`__, or |
| 28 | ++`clustershell <https://clustershell.readthedocs.io/>`__, or |
| 29 | + `slurm <https://slurm.schedmd.com/>`__). For the purpose of this |
| 30 | + tutorial, we will use a single machine and spawn multiple processes using |
| 31 | + the following template. |
| 32 | +diff --git a/intermediate_source/monarch_distributed_tutorial.rst b/intermediate_source/monarch_distributed_tutorial.rst |
| 33 | +index c6bdf66..8542b7f 100644 |
| 34 | +--- a/intermediate_source/monarch_distributed_tutorial.rst |
| 35 | ++++ b/intermediate_source/monarch_distributed_tutorial.rst |
| 36 | +@@ -40,7 +40,7 @@ Prerequisites |
| 37 | + We rely on a nightly build of Titan for this tutorial, so please ensure that other Torch libraries are tracking nightly builds: |
| 38 | + |
| 39 | + 1. **Monarch nightly installed:** |
| 40 | +- `Install script <https://github.com/meta-pytorch/monarch/blob/main/scripts/install_nightly.py>`_ |
| 41 | ++ `Install instructions <https://meta-pytorch.org/monarch/installation.html>`_ |
| 42 | + 2. **TorchTitan nightly installed:** |
| 43 | + `TorchTitan install instructions <https://github.com/pytorch/torchtitan?tab=readme-ov-file#nightly-builds>`_ |
| 44 | + 3. **A valid Titan model config** and **tokenizer** in your working directory |
| 45 | +diff --git a/intermediate_source/rpc_param_server_tutorial.rst b/intermediate_source/rpc_param_server_tutorial.rst |
| 46 | +index 3243316..8fd1588 100644 |
| 47 | +--- a/intermediate_source/rpc_param_server_tutorial.rst |
| 48 | ++++ b/intermediate_source/rpc_param_server_tutorial.rst |
| 49 | +@@ -12,7 +12,7 @@ Prerequisites: |
| 50 | + - `PyTorch Distributed Overview <../beginner/dist_overview.html>`__ |
| 51 | + - `RPC API documents <https://pytorch.org/docs/master/rpc.html>`__ |
| 52 | + |
| 53 | +-This tutorial walks through a simple example of implementing a parameter server using PyTorch's `Distributed RPC framework <https://pytorch.org/docs/stable/rpc.html>`_. The parameter server framework is a paradigm in which a set of servers store parameters, such as large embedding tables, and several trainers query the parameter servers in order to retrieve the most up to date parameters. These trainers can run a training loop locally and occasionally synchronize with the parameter server to get the latest parameters. For more reading on the parameter server approach, check out `this paper <https://www.cs.cmu.edu/~muli/file/parameter_server_osdi14.pdf>`_. |
| 54 | ++This tutorial walks through a simple example of implementing a parameter server using PyTorch's `Distributed RPC framework <https://pytorch.org/docs/stable/rpc.html>`_. The parameter server framework is a paradigm in which a set of servers store parameters, such as large embedding tables, and several trainers query the parameter servers in order to retrieve the most up to date parameters. These trainers can run a training loop locally and occasionally synchronize with the parameter server to get the latest parameters. For more reading on the parameter server approach, check out `this paper <https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-li_mu.pdf>`_. |
| 55 | + |
| 56 | + Using the Distributed RPC Framework, we'll build an example where multiple trainers use RPC to communicate with the same parameter server and use `RRef <https://pytorch.org/docs/stable/rpc.html#torch.distributed.rpc.RRef>`_ to access states on the remote parameter server instance. Each trainer will launch its dedicated backward pass in a distributed fashion through stitching of the autograd graph across multiple nodes using distributed autograd. |
| 57 | + |
| 58 | +diff --git a/intermediate_source/rpc_tutorial.rst b/intermediate_source/rpc_tutorial.rst |
| 59 | +index 791ecf8..93a2259 100644 |
| 60 | +--- a/intermediate_source/rpc_tutorial.rst |
| 61 | ++++ b/intermediate_source/rpc_tutorial.rst |
| 62 | +@@ -33,7 +33,7 @@ paradigms. For example: |
| 63 | + data between observers and the trainer. |
| 64 | + 2) Your model might be too large to fit in GPUs on a single machine, and hence |
| 65 | + would need a library to help split the model onto multiple machines. Or you |
| 66 | +- might be implementing a `parameter server <https://www.cs.cmu.edu/~muli/file/parameter_server_osdi14.pdf>`__ |
| 67 | ++ might be implementing a `parameter server <https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-li_mu.pdf>`__ |
| 68 | + training framework, where model parameters and trainers live on different |
| 69 | + machines. |
| 70 | + |
0 commit comments