Advanced Features
=================

.. note::

   PINNICLE is under active development, and several **advanced**/**experimental** features are listed here:

Mini-batch
----------

PINNICLE supports mini-batch training when using the ``pytorch`` backend. In the current implementation, mini-batches are resampled at every epoch to enhance convergence stability.
To activate mini-batch training, simply add the following setting to the hyperparameter dictionary. PINNICLE will automatically generate mini-batches from the data specified in ``hp["data"]``.
And in the data section, you can set the ``"data_size"`` to a large number or simply ``"MAX"``, to cover a large amount of your data.

.. code-block:: python
   
   hp["data"] = {"ISSM": {"data_size": {"u":100, "v":"MAX", "s":100, "H":10, "C":None}}}
   hp["mini_batch"] = mini_batch

- Avoid setting ``mini_batch`` too large, as this can lead to inefficiencies and excessive memory usage.
- Avoid setting ``mini_batch`` too small, as this can result in wasted computational resources due to the increased overhead of frequent resampling.

Based on our numerical experiments, we recommend choosing a batch size in the range of 500 to 10,000.


Resample collocation points
---------------------------

PINNICLE supports the resampling of collocation points during training. To activate this feature, set the following in your hyperparameter dictionary:

.. code-block:: python

   hp["period"] = 100

This will instruct PINNICLE to resample the collocation points every 100 epochs.


Training with a list of optimizers
----------------------------------
PINNICLE now supports training with a list of optimizers, allowing users to combine the strengths of different optimization strategies within a single training workflow. Users can define an ordered sequence of optimizers, each with its own training configuration, such number of epochs.

.. code-block:: python

   hp["optimizers"] = ["adam", "L-BFGS"]
   hp["epochs"] = [100000, 20000]

.. note::
   This feature is not supported by JAX due to the backend restriciton.


Load Settings from Previous Experiment
--------------------------------------

Each PINNICLE experiment will save the entire settings to the folder defined in ``hp["save_path"]`` if ``hp["is_save"]=True``. The settings are saved into a ``params.json`` file containing all the non-default settings from ``hp``.

To repeat a new experiment with the same settings as in some ``folder_path``, you can easily load the settings by:

.. code-block:: python

   experiment = pinnicle.PINN(loadFrom=folder_path)

However, if you want to run the experiment and save the results in a new folder, you will need to set the ``"save_path"`` and update the experiment:

.. code-block:: python

   experiment.update_parameters({"save_path": new_folder})

Then, compile and train the model.


Load Weights from Previously Trained Neural Network
---------------------------------------------------

PINNICLE can also load the weights from a previous experiment. You can either load the settings from a saved experiment or set up a completely new one. After that, you can load the weights of the neural network from the ``folder_path`` which contains the entire experiment (including ``params.json``, a folder called ``pinn``, and some figures).

.. code-block:: python

   experiment.load_model(path=folder_path)

After loading the weights, compiling and training are the same as for other experiments.


Dummy Equations
---------------

PINNICLE provides :py:mod:`pinnicle.physics.dummy` physics, which allows you to train the neural network using data only. Here is an example:

.. code-block:: python

    dummy = {}
    dummy["output"] = ['u', 's', 'C']
    hp["equations"] = {"DUMMY": dummy}

In this example, we define a ``dict`` with a key ``output``, where the value is a list of three output variables. Then, we add this ``dict`` to ``hp['equations']`` with the key ``DUMMY`` (all uppercase). Additionally, you need to provide the data for ``u``, ``s``, and ``C`` in the ``data`` section, similar to other examples. The neural network will then be trained solely with the provided data.

By default, the ``Dummy`` physics already has ``x`` and ``y`` as ``input``. If there is no need to change this, only the ``output`` needs to be defined.


Manually Override Data Weights
------------------------------

When coupling different physics or observational constraints, the default data weights may not always produce the desired balance among loss terms. Users can manually override the weights for selected variables by setting ``manual_data_weights`` in the hyperparameter dictionary.

.. code-block:: python

   hp["manual_data_weights"] = {"H": 1.0e-6, "s": 1.0e-4}