A Phase 1b/2 Dose Escalation and Cohort Expansion Study of the Safety, Tolerability and Efficacy of a Novel Transforming Growth Factor-beta Receptor I Kinase Inhibitor (Galunisertib) Administered in Combination With Anti-PD-1 (Nivolumab) in Advanced Refractory Solid Tumors (Phase 1b) and in Recurrent or Refractory Non-small Cell Lung Cancer or Hepatocellular Carcinoma (Phase 2)

• For Phase 1b, must have advanced refractory solid tumors in any line of therapy.

• For Phase 2, must have one of the following tumor types: recurrent or refractory NSCLC (any histology), or HCC with elevated alpha-fetoprotein (AFP) ≥200 nanogram/milliliter (ng/mL).

• For Phase 2 only, have had disease progression or be refractory or intolerant to 1 prior line of therapy (first line therapy) for recurrent or refractory for NSCLC or HCC and have refused currently approved second-line therapy. First line therapy is defined as therapy used to treat advanced disease. This may include multiple chemotherapeutic, targeted or immunotherapeutic agents with or without radiation therapy and/or surgery. Each subsequent line of therapy is preceded by disease progression. A switch of an agent within the same drug class (eg, cisplatinum to carboplatinum) within a regimen in order to manage toxicity does not define the start of a new line of therapy.

• For NSCLC:

‣ Prior lines of therapy must include a platinum-based therapy. Investigational agents used in combination with standard therapies are allowed. Participants who received platinum-based neoadjuvant or adjuvant therapy and subsequently received platinum-based therapy as first-line therapy are eligible.

⁃ Participants who have completed neo-adjuvant or adjuvant therapy with a platinum doublet and have experienced disease recurrence within 6 months of completing the platinum doublet are eligible.

⁃ Tumors with driver mutations (epidermal growth factor receptor mutation positive or anaplastic lymphoma kinase fusion oncogene positive) treated with a tyrosine kinase inhibitor or crizotinib are eligible. For participants who have progressed on a tyrosine kinase inhibitor or crizotinib or are intolerant to this targeted therapy, that participant must receive platinum-based therapy prior to enrollment in this study. Documentation of such mutations must be available and entered into the electronic case report form (eCRF).

⁃ Maintenance or switch maintenance therapy after first-line chemotherapy will be considered part of the first-line regimen and is acceptable.

∙ Participants who completed and progressed on a platinum-containing regimen as adjuvant, neoadjuvant, or part of a course of chemoradiation therapy given from locally advanced disease and developed recurrent (local or metastatic) disease within the 6 months before screening would be counted as having received 1 prior platinum-containing regimen and therefore would not require re-treatment with a platinum-containing regimen for Stage IIIB, IV, or recurrent disease and are eligible. However, participants must have received at least 2 cycles of a platinum doublet based chemotherapy before discontinuation for toxicity. If participants received only one cycle of a platinum doublet and discontinue due to clear progression, that regimen should be counted as a prior line of therapy.

• For HCC:

‣ One prior line of therapy which must include sorafenib or participant must have progressed or been intolerant to sorafenib for participants not eligible for transarterial chemoembolization. Participants who had sorafenib for locally advanced disease or are intolerant to sorafenib are eligible. Participants may have had clinical progression only following sorafenib or local therapy.

⁃ Must have Child-Pugh A only. Participants may have any viral status (hepatitis B, hepatitis C, or none).

⁃ Have a viral load \<100 international units/milliliter (IU/mL).

⁃ For hepatitis B participants, must be on a nucleoside analog reverse transcriptase inhibitor (lamivudine, telbivudine, adefovir, tenofovir, or entecavir).

• Have adequate organ function.

• Have a performance status of 0 or 1 on the Eastern Cooperative Oncology Group (ECOG) scale.

• Use an approved contraceptive method.