Is cross-region Kubernetes a thing, and if so, what role does VPN place in this context?

First, etcd is a core component in Kubernetes clusters. It is based on the Raft Consensus Algorithm and a weaker point with it is when the latency between the members (e.g. in the distribued etcd database) in the cluster is larger and vary more. It might work to deploy it with longer latency between the nodes (e.g. regions as in cross-region) but it might not be the best idea, at least not out of the box.
But if you use cross-region deployments, one of the main reason to do it is probably to tolerate a full region outage - but then I think it is even a better idea to deploy independent (e.g. can work without the other) clusters.
What might be interesting in relation to this is Kubernetes Federation Evolution and kubefed.
I don't see VPN (e.g. a single network?) very related to this (this is more its own topic) - at least not when you want independent clusters. SSL/TLS might also be related. And if you do cross-regions, e.g. Software Defined Network or cloud VPC might be related as well. E.g. Amazon VPC is located on a single region, but Google Cloud VPC is global (multi-region).


How to examine Jinja output in Saltstack?

Check out the slsutil.renderer module.

This should do what you want

salt my-minion-id slsutil.renderer /srv/salt/network/init.sls 'jinja'

This module just calls the compile_template function directly for you.

Edit: /srv/salt/network/init.sls is the path on the minion, if you are not targeting the master as your minion, you will probably need to do the following.

salt minion-id cp.cache_file salt://network/init.sls
salt minion-id slsutil.renderer /var/cache/salt/minion/files/base/network/init.sls

or point to whatever file that cache_file spits out.

If you are on 2018.3 or newer, you can just specify salt://network/init.sls


Continuous deployment strategies vs DevSecOps

As you stated, requirements will differ based on company and industry policy; for us, the adjustment from "separation of duties" to automated deployments has been rough. As interim steps, we've started:

Requiring multiple "sign-offs" before deployment; this is accomplished by both reviews for the PR in Azure DevOps and an audit log in Octopus.

Monthly review of deployments by higher-ups.  This is mostly a rubber-stamp activity, but it satisfies auditors.


Bad edit to sudoers file on EC2 instance, how to fix it?

How about user-data ?

I believe adding the "#cloud-boothook" allow to force the user-data to run at every restart.

#cloud-boothook
#!/bin/bash
echo 'test' > /home/ec2-user/user-script-output.txt

If so, you could fix your sudo scripts hopefully... or install / add AWS run commands configuration https://aws.amazon.com/blogs/aws/new-ec2-run-command-remote-instance-management-at-scale/

For sure if it is an EBS Drive, you can detach & fix and then re-attach on the original system. A bit of a pain though.


If you already have the script/tool in place to execute the database migrations then what you might need is to run the tool/script once. AWS ECS allows to schedule the executions of tasks (containers), you can set it up in a way that the container can run just only once.

These ECS tasks can be scheduled via a cron-like job setup or on a CloudWatch event response:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduled_tasks.html


unable to save register value even if the play is skipped!

register will always populate your registered var even when the task is skipped, e.g. you will end up with something like the following:

"kafkaip": {
    "changed": false,
    "skip_reason": "Conditional result was False",
    "skipped": true
}

Since only the commands differ depending on your env, I would store the command in a variable and then execute it, something like the following.

- name: "Set command for env <> QA"
  set_fact:
    get_ip_command: >-
      aws cloudformation describe-stacks
      --stack-name "{{ stack_kafka }}"
      --query "Stacks[*].Outputs[?OutputKey=='KafkaLink'].OutputValue"
  when:
    - stack_kafka is defined
    - ( deployment_env != "QA" )

- name: "Set command for env = QA"
  set_fact:
    get_ip_command: >-
      aws ec2 describe-instances
      --filter "Name=tag:StackName,Values=KafkaCluster"
      --query 'Reservations[].Instances[].PrivateIpAddress'
      --output text
  when:
    - stack_kafka is defined
    - ( deployment_env == "QA" )

- name: Get the Kafka InstancesPrivateIps
  shell: "{{ get_ip_command }}"
  register: kafkaip
  when: stack_kafka is defined

- name: Register kafka_host
  set_fact:
    kafka_host: "{{ kafkaip.stdout }}"
  when: stack_kafka is defined

Note1: there is an output redirection in your QA command which will prevent capturing the output as you wish, I left it out in my example.
Note2: I used set_fact to create the command variable but you can put that in your vars or your inventory to have a leaner playbook.


Legacy build environment in Docker container, incompatible with Jenkins agent

According the the Jenkins docs, you could use your custom docker image as the execution environment, assuming the server Jenkins is running on has Docker installed/running.

pipeline {
    agent {
        docker { image 'your-custom-image' }
    }
    stages {
        stage('Build') {
            steps {
                sh 'commands to run in your docker image here'
            }
        }
    }
}


Local Prometheus and Grafana Installations to Monitor a Kubernetes Cluster

It's possible but after doing many replatforming projects for clients I can't say I would advise it. Have you seen kube-prometheus? It's a one-shot install that will get you 90% of where you need to be for your cluster.

The only way I can advise it would be if you're going to run 5 or more clusters and want central monitoring, but I'm a bigger fan of having a Grafana per cluster.

Edit: Getting started with kube-prometheus is a one command install. kubectl apply -f manifests/


Data Science pipelines and monolithic model blobs

Personally I don't see any reason for which an Artefact Repository - the recommeneded DevOps tool of managing artefacts - wouldn't be applicable to trained neural nets or other artefacts.

The artefact size might have some upper limit for a particular artefact repository, but in such case it would be a technical or policy limitation one, not a fundamental/principial one.

As for applying DevOps methodologies for the process producing these artefacts, I think most if not all of them can be applied equally well, as long as the artefacts:

are produced from some sort of specification which supports change versioning (equivalent to software source code)
are built via a repeatable and automatable process
are validated using some sort of repeatable and automatable verification (similar to QA), eventually using some supporting data (training data in this case, equivalent to DB snapshots, for example)

Side note: monolithic software code delivery is still a big deal and is perfectly maintainable with DevOps methodologies (with a bit of care), not everything can be split in microservices. Size doesn't matter enough to make DevOps not applicable.


Does Buildbot support complicated build/deploy scenarios?

Jenkins supports scripted pipeline which can handle my every requirement.
Buildbot can handle these requirements with custom steps, i suppose, but it's too hard if compared to Jenkins' scripted pipeline.


Cross-Platform Deployment Tasks in Bamboo

There isn't a common protocol that exists on all the platforms for deployment.  However, if you get an SSH server onto the windows platforms then you can use the same mechanisms in Bamboo to do the deployment, and potentially even use the same keys to secure it all.
Windows SSHD - There are a few alternatives (https://serverfault.com/questions/8411/what-is-a-good-ssh-server-to-use-on-windows).  Freesshd you can get administrator level access - been successfully using it with thousands of machines as old as WinXP for years.
The Bamboo SCP task can be used to do the staging to the remote servers - once you have SSH remote access to all your targets.

You can use the Bamboo SCP task to upload files from Bamboo directly to a remote server as part of a Bamboo job. The SCP task is able to copy multiple files and preserves the directory structure for the copied files.

And Bamboo SSH task can then do post deployment install steps, running scripts relevant to the platform.

You can use the Bamboo SSH task to execute a SSH command on a remote computer as part of a Bamboo job.


Is it possible to set up Microsoft Sql Server in a scalable way?

Not impossible, very easy in fact with read-only routing. Clients intending to be read-only specify it in their connect string, so this is per-session rather than per-statement.  You can have up to 9 replicas in an availability group. With modern hardware and storage, that goes a long way to being “scalable” - 10’s of thousands of concurrent connections easily.


Is there a tool that makes it possible to validate Jenkinsfiles online?

Here is some documentation on the Jenkins pipeline linter and its commands. Do you need to validate before a commit? If not, it would be really trivial to run the linting command prior to your pipeline running, and simply fail if it does not pass.

From Command-line Pipeline Linter:

  Jenkins can validate, or "lint", a Declarative Pipeline from the
  command line before actually running it. This can be done using a
  Jenkins CLI command or by making an HTTP POST request with appropriate
  parameters. We recommended using the SSH interface to run the
  linter. See the Jenkins CLI documentation for details on how to
  properly configure Jenkins for secure command-line access.

  Linting via the CLI with SSH

# ssh (Jenkins CLI)
# JENKINS_SSHD_PORT=[sshd port on master]
# JENKINS_HOSTNAME=[Jenkins master hostname]
ssh -p $JENKINS_SSHD_PORT $JENKINS_HOSTNAME declarative-linter < Jenkinsfile

  Linting via HTTP POST using curl

# curl (REST API)
# Assuming "anonymous read access" has been enabled on your Jenkins instance.
# JENKINS_URL=[root URL of Jenkins master]
# JENKINS_CRUMB is needed if your Jenkins master has CRSF protection enabled as it should
JENKINS_CRUMB=`curl "$JENKINS_URL/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,\":\",//crumb)"`
curl -X POST -H $JENKINS_CRUMB -F "jenkinsfile=<Jenkinsfile" $JENKINS_URL/pipeline-model-converter/validate

  Examples

  Below are two examples of the Pipeline Linter in action. This first
  example shows the output of the linter when it is passed an invalid
  Jenkinsfile, one that is missing part of the agent declaration.

  Jenkinsfile

pipeline {
  agent
  stages {
    stage ('Initialize') {
      steps {
        echo 'Placeholder.'
      }
    }
  }
}

  Linter output for invalid Jenkinsfile

# pass a Jenkinsfile that does not contain an "agent" section
ssh -p 8675 localhost declarative-linter < ./Jenkinsfile
Errors encountered validating Jenkinsfile:
WorkflowScript: 2: Not a valid section definition: "agent". Some extra configuration is required. @ line 2, column 3.
     agent
     ^

WorkflowScript: 1: Missing required section "agent" @ line 1, column 1.
   pipeline &#125;
   ^

  In this second example, the Jenkinsfile has been updated to include
  the missing any on agent. The linter now reports that the Pipeline
  is valid.

  Jenkinsfile

pipeline {
  agent any
  stages {
    stage ('Initialize') {
      steps {
        echo 'Placeholder.'
      }
    }
  }
}

  Linter output for valid Jenkinsfile

ssh -p 8675 localhost declarative-linter < ./Jenkinsfile
Jenkinsfile successfully validated.


Secure RDS (private endpoint) and allow connecting from local machine for specific predefined users

I am not sure it will be helped. I have similar design in my office.
I have some Kubernetes Cluster. all node and master running on local ip from a PVC. And my rds too, only have local ip run on same PVC. Kubernetes Cluster already connect to RDS using local ip.
Then, I build VPN server on bastion server. Have local ip and public ip, run on same PVC with RDS and Kubernetes cluster. So when need connect, I just start my vpn connection on my laptop.
Only vpn/bastion server have IP public. + Load balancer too.
I hope this will give you inspiration.


How to scale the control plane of a service mesh?

It's true that when you scale your services is possible to have a bottleneck on the monitoring, but for avoiding that you have to check if your monitoring system is pull or push, in case it is push maybe is not the best option but pull is not the right answer for all situations.

One of the most used monitoring systems today is prometheus, you can se how it works here:

https://prometheus.io/docs/introduction/overview/

But is not the only one and you need to see how it should fit in your stack.

Check the possible integrations with istio

https://istio.io/docs/tasks/telemetry/metrics/querying-metrics/

In case you have already some of that probably need to think on scaling this part instead of the sidecars or istio itself.


Dynamically change config files in c# Application based on environments(dev,QA,Prod) through Bamboo

In your deployment project configuration for each environment you could add a task prior to the actual deployment which would take care of setting up the right configuration file.

It could be a script simply copying/replacing the configuration file just as you currently do manually. The script could be invoked with some pre-configured argument to identify the environment or it could detect the environment via some environment variable configured for each the deployment, see Additional deployment environment options


How to use comments to trigger rebuilds of Bitbucket pull-requests on Jenkins multi-branch pipeline

I personally use the Pull Request Notifier plugin for Bitbucket to trigger my Bitbucket Multibranch Pipeline builds in Jenkins.  Using the GUI configuration tools, you can configure the Pull Request Notifier plugin to trigger a build on any arbitrary PR action, including adding comments.

However, you don't really need to use comments to trigger builds - there is a build trigger button in the PR view in Bitbucket, and you can also use the "Build Now" button in Jenkins to trigger a PR build.  Both of these seem a little bit easier to me than a magic comment.


Multi-Stage Azure Pipeline with a path trigger - first stage always runs

Your pipeline runs for pull requests not because of the trigger, but because of a build validation policy you've set for one or more target branches ("whenever I create a PR to this branch, start this pipeline"). It's the build validation, as part of a branch's policies: https://docs.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=azure-devops#build-validation
You probably didn't specify a path filter in the build validation settings. Here's an example of three different build validations in a single branch policy, all with different path filters:

In short: The path trigger in your YAML has no effect on how a pipeline is triggered for a PR. The trigger you've specified is one that triggers on commits to the master branch, if the commit includes files in the terraform directory.


How to ensure that git subtrees are kept up to date?

GIT hooks could work, but the trouble is that they're not automatically installed when the developers pulls a workspace. Which means they might end up being installed/used correctly or not. Unreliable.

CI scripts could work as well, but the're reactive, simply indicating that something went wrong, in most cases humans still need to identify the culprit (which might be caused by an improper DTO subtree or by something else) and fix the problem.

If you want your developers to properly pull their subtrees - offer them a scripted wrapper to do so - put the effort into that script so that it is simple to use and robust/working well such that the developer's workflow is effectively simplified. If you manage to do this (most of) the developers will see the benefit and they will use it.

For the push side - you need to take into account that mistakes in push will happen (even with the above-mentioned script, if it's not used by all developers, for example). Everyone makes mistakes.

Personally I favour preventing developers from directly pushing their changes into the integration branch and thus blocking possible mistakes from affecting the integration branch.

Instead I'd have all candidate changes (i.e. either pushed in separate branches or just as diffs) funnelled through a mandatory centralized pre-commit verification system. Automated, of course, with the proper git subtrees management made reliable through that automation. The changes would only be committed/merged into the integration branch automatically by this system, if they meet the quality checks.

This approach is applicable to any potential regression-causing mistakes, not just improper DTO subtrees. An example of such approach would be the gerrit-based development on OpenStack.


Structuring Ansible playbooks for low volume, purpose-built images

Unless I totally missed your question, roles are exactly what you need in this case. I'll use examples from my real world. My code below is just to illustrate my point (so it is not fully runnable, might contain errors and is not bullet proof rocket science: you will probably have to adapt/overcome tools limitations). But at least you'll be able to tell me if I got you wrong.

Let's say your servicing a large community of developers on different techs: java, php, nodejs, scala, python, ruby... needing some tools as well like apache, nginx, postgres, mysql, redis...

You have a role for each of these techs/tools, and possibly use existing ones from galaxy that you don't even have to write and that you can download at provisioning time. These roles can possibly be controlled with vars to choose e.g. versions. (see vagrant provisionner config below)

Now, one way you can activate/deactivate roles in a play is with the use of tags

Your unique playbook could look something like this

- name: Vagrant provisionning playbook
  hosts: all

  pre_tasks:
    # whatever you need to do before roles

  roles:
    - role: php
      tags: [php]
    - role: java
      tags: [java]
    - role: nginx
      tags: [nginx,php] # if you always deploy php with nginx
    - role: redis
      tags: [redis]
    # more roles here

    tasks:
      # Whatever tasks are needed to be played after roles

With such a structure, you can now configure your ansible provisioinner in vagrant defining the tags and specifying the vars needed to provision with your single playbook.

Vagrant.configure(2) do |config|

  config.vm define "php_application" do |php_application|

    # Some config for vm
    php_application.vm.provision "ansible" do |ansible|
      ansible.playbook = "playbook.yml"
      ansible.tags = "php,nginx,untagged"
    end

  end

  # Host vars declared globally. You might be able to do
  # it on host by host basis but the doc does not specify it
  config.vm.provision "ansible" do |ansible|
    ansible.host_vars = {
      "php_application" => {"php_version" => "7.2"}
    }
  end
end

Does this fit your needs ?

Note: if a single playbook is too much of a hassle, you can still use the roles and assemble in each playbook the techs/tools you need and reuse things that you will need in several places.


Need help getting documentation of Jenkins

When creating the Jenkinsfile for a pipeline (using groovy) how is the code structure made? Where can I find some examples starting from the easiest to the complex ones?

The files typically ending in *.jenkinsfile can be made in any text editor or IDE. For examples the best place to look is the Pipeline Examples page. There is also a github repo with more examples.

What are the actual differences between stages and nodes on Jenkinks file?

Nodes - are physical machines or virtual machines that run Jenkins as worker. They can be different architectures.
Stages - These are best thought of as the recipe:

Mix the ingredients
Add water
Mix again
Bake

Each stage is an action that Jenkins needs to take.

Where can I learn how to make pipelines from scratch (but not just copy/paste from somewhere, understanding what I'm doing it), i.e.: I want to run a code on the server where I have my Jenkins master every time a fellow coder make a successfully git push

The first two links I gave are the best way of starting but it's purely subjective. I'd suggest starting simple and then building up from there. In the document you referenced there is also a section on Pipelines which might be the missing step for you.


Sending Exceptions to Elastic Stack via REST

It depends on what you're trying to accomplish but IMO if you're interested in reliably getting your exceptions you probably want something between Elasticsearch and your app - a queueing or logging system of some kind like Kafka or Redis.
Send your exceptions there and then drain that to Elasticsearch. Otherwise you run the risk of overwhelming ES's ability to accept writes at the very time you need it most (cascading failure, DDOS etc).


The way I'm understanding your question is that you're writing a module that takes an optional vpc_id as an argument, and either uses it if specified or creates a new VPC itself if not specified. You then need to use the VPC id elsewhere in your configuration, regardless of whether it was provided by the caller directly or if it was created by the VPC module.

If I understood that correctly, here's one way to get that done.

First, we'll declare the vpc_id variable for your own module that the rest of this will use to make its decisions.

variable "vpc_id" {
  type    = string
  default = null # optional with no default value
}

From this we will derive our decision about whether to create the VPC, which I'll write as a local value here mainly for exposition purposes but you might choose to inline it at the point of use in practice depending on what feels most readable for you:

locals {
  create_vpc = var.vpc_id != null ? true : false
}

We can now use that value both with the module and with a data resource so that we'll either create a new VPC or retrieve data about the existing one:

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  create_vpc = local.create_vpc
  # ...
}

data "aws_vpc" "selected" {
  count = local.create_vpc ? 0 : 1

  id = var.vpc_id
}

We can then write another local value that chooses which of those two locations to derive other VPC-related values, such as the id and cidr_block from:

locals {
  vpc = (
    local.create_vpc ?
    {
      id         = module.vpc.vpc_id
      cidr_block = module.vpc.vpc_cidr_block
    } :
    {
      id         = data.aws_vpc.selected.id
      cidr_block = data.aws_vpc.selected.cidr_block
    }
  )
}

You can then use local.vpc.id and local.vpc.cidr_block elsewhere in the configuration to get the appropriate values to use regardless of whether this module created its own VPC or not.

The above is the most direct answer to your question. I also wanted to share a different approach that is based on the patterns described in the Module Composition section of Terraform's docs, which might be appropriate for you if you're building a shared module intended to be called by another Terraform module.

A different way to approach this problem using composition is to say that your module never creates its own VPC, and always expects one to be provided to it. This is an example of the dependency inversion principle. Here's how that might look for your particular use-case...

In your module, you can declare an input variable called vpc that has the same structure as the final local.vpc that we created in the previous approach:

variable "vpc" {
  type = object({
    id         = string
    cidr_block = string
  })
  # This time the VPC is required. The caller decides how to obtain it.
}

Your own module's expressions would then get the VPC id as var.vpc.id instead of local.vpc.id as in the previous approach.

The calling module can then decide between several different ways of providing the value for that variable:

# Using the terraform-aws-modules/vpc/aws module
module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  # ...
}

module "your_module" {
  source = "./modules/your-module"

  vpc = {
    id         = module.vpc.vpc_id
    cidr_block = module.vpc.vpc_cidr_block
  }
}

# Directly using the aws_vpc resource type

resource "aws_vpc" "example" {
  # ...
}

module "your_module" {
  source = "./modules/your-module"

  vpc = aws_vpc.example
}

# Using the aws_vpc data source to find an existing VPC

data "aws_vpc" "example" {
  tags = {
    Environment = "STAGE"
  }
}

module "your_module" {
  source = "./modules/your-module"

  vpc = data.aws_vpc.example
}

In other words, this lets your new module focus on whatever new problem it is trying to solve while being decoupled from the problem of creating a VPC, allowing the calling module to obtain that VPC information in whatever way is appropriate. The appropriate way to obtain the VPC might change over time, and this composition approach means that callers can have more control over how they manage the transition between different approaches, independent of the implementation of your new module.

Module composition isn't always the best option, but I wanted to point it out because the Terraform documentation specifically recommends considering this approach and recommends against having many levels of nested module calls.


OK. The answer was simple. I needed to template the item.path. I guess this is a beginner mistake.
- hosts: localhost
  tasks:
    - name: Find ini files
      find:
        paths: /home/myuser/projects/infra/php
        file_type: file
        recurse: Yes
        patterns: "*php.ini"
      register: files_matched

    - name: Debug files_matched
      debug:
        var: files_matched.files

    - name: Debug files_matched loop
      debug:
        var: item.path
      loop: "{{ files_matched.files|flatten(levels=1) }}"
      loop_control:
        label: "{{ item.path }}"

    - name: Ensure "sendmail_path=/opt/mailhog/mhsendmail is in section "[mail function]"
      ini_file:
        path: "{{ item.path }}"
        section: "mail function"
        option: sendmail_path
        value: /opt/mailhog/mhsendmail
        mode: 0644
        backup: no
      loop: "{{ files_matched.files|flatten(levels=1) }}"
      loop_control:
        label: "{{ item.path }}"


kubernetes nodeport service load balancing

service is a cluster-wide object. In each node, kube-proxy will create iptable load balancing entry with all available pod's IP based on selector for a service. So NLB will distributed traffic evenly among all nodes. Then each will loadbalance the traffic to all available pods using round robin algorithm.
We will have those traffic distributed among all pods like below.
p1 -> 33.3330%
p2 -> 33.3330%
p3 -> 16.6665%
p4 -> 16.6665%
This will deviate a little bit.


yarn start should be the CMD of your docker image (I.e what is run when you start a container) and not a RUN command (which is run when creating the image used to satrt a container later)
From your exemple, something like this should be better:
 FROM node:12-alpine

 WORKDIR /app
 COPY ./server /app

 WORKDIR /app/server
 RUN yarn
 CMD yarn start


Condition CircleCI build on what part of monorepo changed?

This discussion gives a pretty good suggestion on how to accomplish what I want.

https://discuss.circleci.com/t/does-circleci-2-0-work-with-monorepos/10378/11


DevOps has three parts: Tooling/automation, Organizational, and Cultural.

You're basing your organizational structure and process' on your business' maturity.  Which is smart.  My experience in a startup and with a fortune 1000 company has afforded me the experience to denote that starting with DevOps is better for company growth.  It's efficient and fits nicely with agile.

Tooling

Being your maturity is really young, you should buy your DevOps process as saas products to reduce time to market.  Instead of Jenkins use circleCI or similar product.  Instead of VM's and docker, use Heroku.  Try and incorporate the toolset but do it in a manner that works for the size of your business.  If deploying code manually takes 4 hours a week and automating it in 1 hour saves that.  Do it.

Organizational

Also from an organizational standpoint, you will naturally start with all developers as the responsible party for operations/features even product development.  Take a look at http://web.devopstopologies.com/  for organizational structures.  Our startup started with type 2, and as we mature will move toward type 7.

Cultural

DevOps is simply teamwork.  Work together and everything gets easier.   It usually is a problem because leadership doesn't keep this in mind so as you start out, ensure that everyone is apart of the business.  This is also reflected in agile.

So in short, yes pursue DevOps but do so at the level that meets your business maturity.


Sending HTTP requests to private EC2 DNS from other EC2 host works, but not from container on EC2 host

There are a couple of things that you can rule out, thanks to the testing that you have already done.  You know your container port has an active network socket, that is listening for and accepting connections.  You also know that your EC2 instance has working networking.
I can think of three obvious problems:

Your host is using local DNS resolution to identify your hostname as 127.0.0.1 and your container port is not listening on the loopback adapter. you can test this on the server by pinging your DNS name and see which IP address it pings.  You can also use telnet to connect to the port on 127.0.0.1 and see if that port accepts connections.  To fix this, you may choose to make listening on the local loopback work, but you could also fix this by using a DNS entry other than the hosts AWS generated name.

You are resolving the hostname to its private IP address, which then is filtered by the AWS Security Group. On your security group you have enabled incoming connections to this port, but you have specified the source either as a cidr that doesn't include its own IP address, or as security group that is not attached to this EC2 instance.  This is easy to fix in the security group editor.  To prove that this is working successfully and allowing the connection, you can set up an AWS Flow log which will allow you to see all TCP connections on the Network interface.

The Container hosting which provides the port mapping also has a firewall element that restricts incoming connections to only some IP addresses, and may also provide some diagnostics to assist in understanding whether it receives the connection, and whether it decides to route or reject the connection to the container.

If it is less obvious, i think you may be in the world of using IP Tables to generate log messages in the syslog as your connection routes though the IP stack.  This will at least allow you to confirm that at a network level you are generating a TCP connection and receiving the connection within the OS.


What is a Notes Day and can it be used in DevOps transformation?

I see some interesting parallels in this story and The Phoenix Project.  (Spoilers ahead for the book, if you haven't read it read this anyway :))

I take it the title for "Notes Day" comes from the term of art in Hollywood, where "notes" are constructive criticism about a work.  In this story, Catmull plays the part of Parts Unlimited CEO Steve Masters.  Catmull accepts that he can learn from his employees how to do a better job, as Masters does in Phoenix Project when he does his public mea culpa and apologizes to Bill Palmer.

In a sense, Catmull is also Brent- he's a major constraint.  If his time was so precious, it meant that Pixar was struggling to delegate authority.  It seems  Catmull was critical to every decision, rather than able to set large-scale goals and leave it to employees to figure out how to achieve them.

An event like this might work well at some companies, less well in others.  I think it's the key idea behind Notes Day that is most valuable: accountability and transparency from the highest levels of management to the last employee.  That value can be spread throughout a company in a variety of big and small ways.


How to visualize dynamic microservice architecture running with Consul?

I recommend that you take a look at the NetworkX library for Python.  With a script you could query Consul and build a graph of linked nodes which could be rendered to an image file and served with a web server.  A Flask microservice for this would be fairly simple to set up and provide a very useful visualization of your microservices spiderweb.


Deploy to GKE from CI server outside of cluster

The easiest approach does appear to be to use gcloud, as trying to get a kubeconfig without it proved tricky.

Fortunately, gcloud, kubectl and helm are available as a single docker image kiwigrid/gcloud-kubectl-helm. For example:

docker run -it --rm --volume ./gcp-key-file.json:/data/gcp-key-file.json:ro kiwigrid/gcloud-kubectl-helm:2.11.0-224.0.0 bash
gcloud auth activate-service-account --key-file=/data/gcp-key-file.json
gcloud container clusters get-credentials dev-cluster --project <project name> --zone <zone>


How to setup multiple build environments in Azure DevOps/TFS2018?

This is a general issue with react apps that is independent of what CI/CD stack you use that is discussed at https://stackoverflow.com/questions/49975735/rendering-an-environment-variable-to-the-browser-in-a-react-js-redux-production

With the Azure family of CI/CD products it is expected you create a release artefact then release that same artefact into environments using a release workflow. That is a typical best practice and expected by ”enterprise” release management processes. The general manifesto of native cloud apps is 12factor.net, and it also advises to ”build once, deploy many times, configured by environment variables”. So its a bit of a surprise to me that the best practices isn't really known in the react community so you have to hunt around for solutions.

In my case at that link above I didn't want to make deployments of our react apps different from our backend microservices where we have ”build once, run anywhere”. So we chanced the js code to fetch env vars from the server at page load. Also with this approach and by using containers we are guaranteed to always run the same front end libraries everywhere even if the front end build tool is misconfigured or buggy in terms of not always using the exact same dependency on every run.

In your case you do have a release workflow where you can use all the same built-in and marketplace tasks of a build workflow. This means its straight forward to add a front-end compile task to the release workflows.

On the downside a frontend build of a large react app can take a lot of time and a lot of memory. Memory probably won't be an issue for you. The time taken will be a little annoying. Yet the approach wouldn't be invasive so I would recommend it may be the least effort to set up and the easiest for colleagues to troubleshoot and maintain in the future as long as you have a stable frontend build. Historically npm was less reliable than other frontend builds at giving the exact same lib versions every run for the same package.json (e.g., npm vs. yarn). So you need to study that aspect and ensure that you are certain that every time you run the frontend build it is guaranteed to use the same exact dependency versions.


Can you remove a commit made to a branch on AWS code commit?

Simplest answer is to just revert the commit.  AWS Code Commit is standard Git and will support everything Git allows you to do.
I.e. find commit hash:
git log -n1
commit 444c954e458cc446e2a7a1f2659adf71bdf55580 (HEAD -> my-branch, origin/my-branch)

Then do the revert:
git revert 444c954e458cc446e2a7a1f2659adf71bdf55580
git push

It'll revert your repo to whatever state it would be without that specific commit.  Note that if you have other commits that modified the same files afterward, you will have a merge conflict and will have to fix them manually.


I'm not 100% sure I understand from the question, but it sounds like the Docker solution would be to go from having an (physical?) appliance with an OS and your app installed on it, to having an appliance with an OS and Docker on it, running a single container with your app in it. That doesn't obviate the need to update the OS in the host, and it adds a layer of complexity (and more updates to contend with, as you'll now have to keep Docker and the OS patched) with no readily-apparent benefit as far as the specific areas mentioned in the question are concerned.

However, if you're talking about going from a virtual appliance to a Docker container, that could potentially smooth things out for you, but it also adds Docker as a dependency for your product; you're shutting out anyone who isn't using Docker and doesn't want to add it to their stack just to use your product. You could continue to support those that don't/won't use Docker by continuing to ship the (now "legacy") virtual appliance as before, but now you've just doubled your workload because you have two distributions to support instead of one.


Generally, when referencing 3rd party packages (Docker base images, Maven dependencies, Ruby Gems etc.) it is worthwhile to be as specific as possible. You want to avoid randomly upgrading dependencies at a later time, just because there's a newer version out there, just because you only specified a prefix of the actual 3rd party release number.

I personally would do whatever it takes to make sure that I get the same upstream version on every build, forever, and that upgrading always is an activity I have to d, that is never done for me. For example, in Ruby, the bundler keeps specific versions in the Gemfile.lock. While the Gemfile can specifiy vague versions (like "~> 3.1" giving you anything ">= 3.1 and < 4.0"), whenever a version is picked by the automatic resolving, the actual immutable version is fixed in the Gemfile.lock and will never change on its own.

The reasoning is very simple - we use tools like Docker specifically to get out of the upgrade/version hell we had in the past; to make sure that we have the same versions on all environments, and to make changes visible (in the form of git commits etc.). Introducing the same all over, by having "floating" upstream version numbers, would really be a shame.

There is really nothing elaborate like that Ruby Gemfile.lock, for Docker base images, so in this case you have to look at the images you are pulling; they should document which tags are considered permanent.  Then just specify those. And stay clear of latest, obviously.

For your particular example, to pick at random, a version like 3.6.6-alpine3.7 seems to be pretty fully specified. I'd expect not much more than bugfixes from upstream. But if I were especially paranoid on that day, I might specificy a @digest for the FROM (I'm not that paranoid normally...).


Referencing current variable in Ansible conditional statement?

How about something like this:

hosts: all

vars:
  user_servers:
    user1:
      - testing1

  users:
    -name: user1
    -groups: '{{ user_groups }}'
    -state: present

roles:
  - name: Add filtered users
    tasks:
      - name: Filter users
        debug: User {{ item }}
        with_items: "{{ users | default([]) }}"
        when: {{ inventory_hostname_short }} in user_servers[item]
        register: filtered_users

      - name: Add filtered users
        include_role: wilshersystems.users
        with_items:
        - {{ filtered_users }}
        loop_control:
        - loop_var: users


The Amazon Cognito streams feature can be used to backup data.

Currently, Amazon does not provide a solution to backup their Cognito user Pools. You can use the following NPM package called "cognito-backup":

Install:

npm install -g cognito-backup

Usage

cognito-backup backup-users <user-pool-id> <options>  Backup all users in a single user pool
cognito-backup backup-all-users <options>  Backup all users in all user pools for this account

Examples

cognito-backup backup-users eu-west-1_1_12345
cognito-backup backup-users eu-west-1_1_12345 --region eu-west-1 --file mypool.json
cognito-backup backup-all-users eu-west-1_1_12345 --region eu-west-1 --dir output

Source: https://www.npmjs.com/package/cognito-backup


Can a 5 US $ VPSE support VAD stack with 2 relatively-small CMSs?

In this scenario, the cpu cycles required to read from the database and render to PHP will all be handled by a single core, ie they will be handled one after the other. This will be about the slowest way to deliver website traffic out of your server.

Each time a request is made, a certain amount of memory is occupied by PHP. You can test the actual amount, but you can only ever be sure it is not greater than memory_limit. Based on a use of no more than 256MB per Drupal site request, you can divide your 1GB of memory by that number (minus some memory for apache, MySQL and the OS). The result are 3 requests fit into memory only. You certainly do not want to allow more, because either your system runs out of Ram or it will use swap disk space.

The use of swap disk space is even worse, as it will slow down requests even more and consumes cpu itself.

3 requests can be used up quickly by some Ajax requests in the page or your Admin backend, either one likely bringing the server to its knees. It would be best to check in Chrome inspector how many PHP requests the site makes on frontend and backend.

Your situation will improve somewhat if you can dial down your PHP memory limit to say 64mb, but you still have the single core as the bottleneck.

So the key to keep this stable is to reduce the number of requests allowed either in apache or php-fpm to a number multiplied by PHP memory limit that stays smaller than your 1gb memory.

Your best option would be to copy out your Drupal pages to a static website every time a change is made. A transparent way for this is to put a cdn in front of your website. However you need to be sure.

Another non technical practical solution is to quadruple your hosting budget and find a sponsor or side gig to pay the charges.


Looking at a Career jump into DevOps and advice or suggested reading

I do not think that there is a single correct piece of advice for such a broad question, but here are my thoughts your situation:

As a starting point look at your current workflow, processes, and technologies being used at the company and then research some tools
(Jenkins, Kubernetes, Docker, Grafana, Ansible, Kubana etc)
or techniques(writing your own scripts or programs) that might be useful to your company or any projects you are working on.

I would also suggest maybe attending local DevOps meetups or talks in your area for more exposure.


Docker Sign an Image with an existing Certificate

This documentation indicates that it is possible to add collaborators by trusting their key and that it is also possible to revoke access.

Having said that, the answer to:

  either an image is signed or it is not, but it is not qualified by who
  performed the signing

that it is possible to check who signed the image if every collaborator creates their own key and each key is trusted individually.

As notary and dtr were already mentioned, an alternative is Portus. Portus is able to restrict access to docker images, e.g. some users are only allowed to view images, while others may pull them. It is possible to revoke access as well.


Rancher & gitlab-runner - is there a way to get the gitlab-runner to tell Rancher to start the CI test environment instead?

The short answer is "it depends". Rancher itself runs using system-docker and the docker instances it starts for you, such as gitlab-runner are run separately in regular docker. Gitlab-runner then will start runners depending on its configuration:

  There are three methods to enable the use of... docker run during
  jobs; each with their own tradeoffs.

https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#runner-configuration

My eventual solution

I'd initially installed Rancher 1.6 and just couldn't get the dependencies working (Rancher OS running Rancher 1.6 and Rancher agent running on a couple of Alpine nodes).

The upgrade to Rancher v2 was a steep learning curve as you need to learn the Kubernetes ecosystem tooling (helm, kubectl, etc). I still couldn't get a gitlab-runner working under Rancher v2, but upgrading to v2.0.8 fixed it. I could add the custom helm chart for gitlab-runner and after cribbing the settings from https://gitlab.com/charts/gitlab-runner/blob/master/values.yaml I could get gitlab-runner working under Rancher v2.


What are the pros and cons of running application testing during vs. after a docker build?

I support J.Doe's suggestion for separate Docker files for each of the build and test stage. Such approach also allows you to:

re-run the test stage for whatever reason (known intermittent failures, for example) without re-doing the build stage
run multiple different test stages in parallel using the same build - for example instead of running a single long test stage made up of serialized tests, for a potentially significant overall pipeline speeding up.


GUI over GitLab CI Environment Deployments With Multiple Instances?

Check out the GitLab Operations Dashboard feature (https://docs.gitlab.com/ee/user/operations_dashboard/)

There is a screenshot on that page, too, so you can get an idea of it.


I think you're asking too many questions in one post for the StackExchange QA format, but I can address the Bitbucket-related questions.

For automatically setting up jobs for each branch (and PR) in a Bitbucket project or repository, use the official Bitbucket Branch Source plugin.  This will also take care of the build status aspect of your question.
If you are using Bitbucket Cloud, the Bitbucket Branch Source plugin will also take care of creating webhooks in order to automatically trigger builds.
If you are using Bitbucket Server, you will need some other way of triggering builds.  I personally use the pull request notifier plugin for Bitbucket since our workflow is PR-based, but you can use whatever plugin or tool works for you - there are a ton of them out there.


How can I achieve better load balancing of different systems on a Jenkins?

The default behavior of Jenkins is to use the same node over and over unless it's unavailable. If you use labels you can target resources.

Take a look at plugins like Least Load and Scoring Load Balancer to help distribute the load. CloudBees also provides an Even Scheduler plugin as a commercial offering.

Kubernetes (e.g. Kubernetes Plugin) to schedule ephemeral agents could also satisfy your distribution needs.


Ah I worked it out now.  The epp call needs the module name in it:

content => epp('moduleName/file.epp')


Automate creation of branch + pull-request

You want a RCS/VCS that handles the version incrementally for you. I'm not sure how this relates to CI as the idea is to Continuously Integrate instead of spawning more branches. I'm not sure what you're using for your PR's, so I'll use Atlassian Bitbucket as an example. In the response to pushing up a new branch you get back the url for creating a PR which has an API which you can call to create your PR.
Git commands will do the branching for you:

# Get the latest description, usually based on tags : https://git-scm.com/book/en/v2/Git-Basics-Tagging
$branchName = git describe
# Checkout the new branch
git checkout -b $branchName
# Push up the branch to origin
git push --set-upstream origin feature/$branchName

It's best to have a firm grasp of git when dealing with VCS.
* https://try.github.io/
* https://git-scm.com/book/en/v2/Getting-Started-Git-Basics


How to fetch resource group name( with pattern matching) using Azurecli

There are at least two ways to achieve this.

First is to go with transforming json output to proper object for PowerShell as described in this thread:

how do you select just one property from an array of “results” from the Azure command 'az group list in powershell'?

Second it to use --query global parameter for Azure CLI:

az group list --query "[?contains(name,'kafka')].name"

But remember to use at least PowerShell 6.2 as previous versions have an issue:

powershell --query error #9047


Running AWS lambda in serial (for a given parameter)

Use SQS with Lambda to accomplish this.
SQS supports 2 types of queues: Standard & FIFO. FIFO is appropriate for your use case. Create a FIFO queue & attach a Lambda function trigger to it. This causes the Lambda function to be triggered when a message arrives in the SQS queue. When you send a message to a FIFO queue, set the userId as the message group ID. This will ensure that messages from an individual user are only processed serially but messages from different users are processed in parallel.
Here's the relevant section from AWS Lambda documentation:

For FIFO queues, Lambda sends messages to your function in the order that it receives them. When you send a message to a FIFO queue, you specify a message group ID. Amazon SQS ensures that messages in the same group are delivered to Lambda in order. Lambda sorts the messages into groups and sends only one batch at a time for a group. If the function returns an error, all retries are attempted on the affected messages before Lambda receives additional messages from the same group. Your function can scale in concurrency to the number of active message groups.


Jenkins hangs on Windows CMD build that has GUI

My current, less ideal solution to this is to run Jenkins in a terminal logged in to a desktop rather than as a service. I'd rather not have to do that but it has solved the problem entirely.


Running task or role before any playbooks

What you are looking for is the meta/main.yml in the roles of Ansible.

So make your every playbook (the ones you want to run after a task/role) as a role itself (to make it secondary execution) and what ever task/role you want to run before it should be added to the meta/main.yml of all the secondary execution roles. This would make all your roles execute the role mentioned in meta/main.yml


How do I run the InSpec using a wrapper profile?

I stumbled across the answer.  From within the profile directory, run this command:

inspec exec .

Don't forget to set the depends key in inspec.yml.


Aws autoscaling group in private subnet, private communication?

I am assuming that the backend application has been set up in its own subnet that has been configured for private IP addresses only.  If not I would recommend that you do that as a starter for 10.

With that in place, you could set up an internal application load balancer to work on private only and have your backend ASG use this.
from the AWS docs
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-application-load-balancer.html


In k8s, do the controller nodes have to be in same VPC as the worker nodes for a cluster?

For Amazon EKS, the control plane is running in a separate VPC that is managed by Amazon. You only need to care about the VPC for the worker nodes.

Amazon EKS control plane – Deployed and managed by Amazon EKS in an Amazon EKS managed VPC. When you create the cluster, Amazon EKS creates and manages network interfaces in your account that have Amazon EKS  in their description. These network interfaces allow AWS Fargate and Amazon EC2 instances to communicate with the control plane.

From Amazon EKS Networking


Git Repository Manager conditionally deny pull request

Most repository managers have some way of implementing this.

GitHub (and GitHub Enterprise) has the idea of statuses on a commit; you'd set up a webhook that pings your tool's webserver when a commit is pushed, and then it would use the status API to set whether or not it passes the version check.

Bitbucket, when self-hosted, supports plugins to integrate nicely with it, including ScriptRunner, which can be used to do all sorts of things similar to what you want.

I'm not familiar with GitLab, but it probably has some similar method.

You will need to do some programming to implement the checking logic, but these pieces should allow you to kick that off when appropriate and use the results to block the merge.


Apply resilience-testing in a company (to promote DevOps)? If true, why, how and what effect does it have on the Engineers? If false, why not?

I think that chaos testing will promote DevOps unless people have been informed in advance and proper alert management has been enabled. If people do not get triggered then this will not work.


CI/CD free for hobby side projects. Jenkins on Heroku as alternative, with ephemeral dyno

You should try CI / CD like:

Codeship ( free tier, and totally free for open source projects )
travisci
circleci, ...

Openshift, as @JamesKnott mention, but this one is a full PaaS, so it is meant to host your app too.

Hope it helps


The key words here are:

  Pipeline on Bitbucket

You already have a pipeline. This is where you would stick the deployment command.

AFAIK there is no standard tooling for registries to 'push out' any changes; you'd have to code a custom integrator that continually polls for changes and compares. This would be a lot of work. Same goes for having your servers directly listen for changes on the registry.

It would be far easiest (and is the standard) to simply have your build pipeline issue the deployment command after its successfully pushed a new image to the registry.

There are benefits to this:

your pipeline is aware of whether the push to the registry failed or passed (and you can handle that failure gracefully with retries)
your pipeline will be aware of the new registry tag
your pipeline can control exactly which tag gets pushed where, when, and how quickly
your pipeline has the same insight to whether the deployment passed or failed, and can attempt retries or perform notifications of failure


Practical reasons for 200 limit of AWS resources in CF templates

The 200 limit can be raised, however this is a grinding process that can take months. You work through customer support to get "elevated" as a client to more AWS privileges.

Resources:
- Worked under a company that specifically ran into this problem, took 8 months before that limit was lifted.


How do I use only an encrypted version of my AWS RDS password in my terraform code?

If I am understanding your question correctly, what you are describing sounds like a perfect use case for SSM. You can store your RDS password as a secure string parameter encrypted by KMS, then reference it in you terraform file as a data source with decryption. With this method, you only have to reference your decrypted password, and never directly store the encrypted or plain text in your tf file.

data "aws_ssm_parameter" "foo" {
  name = "foo"
  with_decryption = true
}


Is it reliable to use Docker's hash as a mean to determine image's version between different platform

Yes, you can use the repository digest for this. Note that while this is a hash of the container, it's a hash specific to the repository, and is NOT the image id, which is a separate sha256 hash.

Once you push or pull a docker container to/from a registry, it acquires some repository metadata, including the repository digest for that container. You can see that digest by running docker inspect <container-tag-or-id> and looking for the RepoDigests section of the json (or just pipe that command into jq '.[].RepoDigests' if you have it installed.

Now you can compare that with the docker pullable of the kubernetes pod you want to check by running this command:

kubectl get pod <POD> -o jsonpath='{.status.containerStatuses[*].imageID}'

If the hashes of both of those values match, then you can be sure they are running the same container.


Where should we keep application properties file while doing build promotion using Jenkins?

I disagree with @levi, you should stick to the 12 factor app method. Leaving configs in the repo scusk. Simple changes to a password or a small setting requires a commit and a pipeline rebuild. That's too much work for something so simple.

The last 3 companies I've worked for are using 'single branch' repos. There is only the master branch, and developers create 'feature' branches off of that. All the different environments are built from master.

In that case, we have a static config file, but all the values are injected by the build pipeline. You can pull them in from different sources: build server env variables, an external tool (Hashicorp Vault), or even another repo that has all the configs (this last one is least desirable).


What is, or could be the formal and/or officially established definion of a "stage"?

Often in technology we co-opt English words and give them alternate meanings. However, in this case it's really just the standard definition:

  A level, degree, or period of time in the course of a process


Recommended SVN branching strategy for DevOps

We use Git for our VCS. We follow Trunk based development strategy.
Trunk-Based Development (TBD) is where all developers (for a particular deployable unit) commit to one shared branch under source-control. That branch is going to be colloquially known as trunk, perhaps even named “trunk”.
 lie of omission

This trunk is always stable, but it can have unfinished features covered by feature toggles. This type of development works

When you are just starting up
When you are iterating quickly

In short, feature branches will give you better isolation, but require you to deal with the pain of deferred integration, and merges. Toggles give you continuous integration, but require you to design/implement your code in such a way that supports toggles, and introduce the risk that unfinished feature code could negatively affect production.


Get all Deployments referencing given ConfigMap by the ConfigMap's name

You can use kubectl get -o json and parse the output using jq to iterate over the deployments, matching any desired configMapKeyRef.name and returning a uniq list of deployments.
This should work:
kubectl get deployments --all-namespaces -o json | jq -r '.items | map(select(.spec.template.spec.containers[]?.env[]?.valueFrom.configMapKeyRef.name == "london-tz-config" ) | .metadata.name) | .[]' | uniq


How can I retrieve the execution status of parallel triggered child jobs to a pipeline script

Use the result property of object returned from the build step.

This was already asked and has an answer over on StackOverflow.  Additionally, this is hinted at in the official build step documentation in the description for the propagate parameter ("use the result property of the return value as needed").  I haven't tested it as I don't have a test Jenkins instance available right now, but based on those two pieces of information it looks like something like this should do the trick:

def b = build('my-job-name', wait: true)

def r = b.result


How can I check when a subnet was created?

On CloudTrail, select Event history, choose Event name from the first drop-down, and CreateSubnet from the second. Then fill in your timeframe.
If you don't have CloudTrail enabled, you will probably be out of luck, as I don't see a timestamp field available via the EC2 API.
References:
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-a-trail-using-the-console-first-time.html
https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-subnets.html


How do I get the hosts from a specific section of an Ansible `hosts` file in a Jinja2 Template

The magic you need is magic variables, in particular the groups variable, which is

  a dictionary/map with all the groups in inventory and each group has the list of hosts that belong to it.

Then you can use template filters to format the list: use map with regex_replace to add the quotes to all list elements and then join them into the final result.

{{ groups['elasticsearch_nodes']|map('regex_replace', '(.*)', '"\\1"')|join(',') }}


Is there a way to tell if an Azure Devops Extension is secure to use?

Can't comment yet, but most extensions host their source code publicly. So you could read the source code to check if something is going on that you don't want.

If the source code is not available, you could check the company for certain certifications like ISO 27001 or the like so you know they get audited.


I trying on Windows 10, Trying to connect / login to work interactively on container but I'm getting this kind of error

Restart docker, it falls into this state after the OS reboots.

The below thread mentions running it a swarm as preventing the issue from reoccurring and several ways to resolve the underlying OS issue with the software that may, or may not be workable for you.

https://github.com/docker/for-win/issues/1038


Passing parameters from Jenkins to Ansible playbook

You can use -e switch for that purpose:

ansible-playbook -e 'foo=${FOO}'

where the FOO can be defined in the Jenkins envirionment variables. If it is defined within the parameters block of your Jenkinsfile, then you need to use ${params.foo}. Then, the value can be accessed by the Ansible playbook with {{ foo }}.


Improving deployment process of agent software

ansible-pull would work well for this, though it makes the assumption that your clients are using ansible. That's not an assumption I would be comfortable making in your shoes.

At this point it would appear to me that developing our own script to further automate the process would be a better solution

I agree with that. Since you already have a script to figure out what needs to be downloaded, I'd extend that script to download the needed file(s) and trigger the installation/registration.


How do you monitor status of multiple docker containers?

Note: answer based only on docs, I didn't play with docker much.

If you choose scripting you could pick a higher layer than just ps, based on either:

the docker-compose events CLI:

Usage: events [options] [SERVICE...]

Options:
    --json      Output events as a stream of json objects

  Stream container events for every container in the project.

the docker events CLI:

  Description

  Get real time events from the server

You'd be looking for the die event of the container(from Extended description):

  Use docker events to get real-time events from the server. These
  events differ per Docker object type.

  Object types

  CONTAINERS

  die

The event appears to be generated regardless of the restart policy (from Docker Events Explained):

But you're right, there's a ton of monitoring tools out there, I also suspect some already do what you're seeking.

Just aimlessly wandering through those links I got to Implementing Docker event monitoring from scratch:

  What else? I can create alerts on them. The next time any of my
  containers decide to puke and die, I can get an SMS message about
  their doomed state.


It looks like that the official Consul documentation is more up to date than the link that was posted in the question.

If one starts with the Getting Started guide, then one could start a consul agent in no time and the documentation also explains how to proceed.

user@localhost /tmp $ ./consul agent -dev
==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.0.6'
           Node ID: '777c5475-bc78-6e38-75a6-42038eef7cb8'
         Node name: 'localhost.localdomain'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
      Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2018/04/09 21:33:04 [DEBUG] Using random ID "777c5475-bc78-6e38-75a6-42038eef7cb8" as node ID


What are the tools you recommend for deploying / automating / organizing vm in the private cloud

The answer will depend on how much you want to change. I will describe how most larger companies solves this nowadays.

Containerize the apps. Run them in Docker containers. And use a container orchestration tool, like Kubernetes to manage deployment. But to do this your application must probably be redesigned to follow The Twelve factor methodology.

Both Amazon, Azure and Google have Kubernetes cloud services. If you want to continue with private cloud, OpenShift is the most popular Kubernetes-product.


"Software Development" which environment is it: Manufacturing or Projects?

According to Wikipedia

  Manufacturing is the production of merchandise for use or sale using
  labour and machines, tools, chemical and biological processing, or
  formulation.

and

  a project is an individual or collaborative enterprise, possibly
  involving research or design, that is carefully planned, usually by a
  project team, to achieve a particular aim.

Based on the definitions Software Development could both be seen as Manufacturing and a Project.

It could be seen as manufacturing from a deliverables perspective as Development activities mean that that the software is created, but does not guarantee quality. Therefore QA is required and to get it online Operations tasks are needed as well. In summary, Software Development could be seen as manufacturing as multiple specialisms are required to make the feature available for a customer.

It could also be compared with a project as the Dev team itself does R&D and creates a (software) product.

From a DevOps perspective I would advocate to compare Software Development with manufacturing rather than project as DevOps is about let work multiple specialisms together and not about creating silos like project.


How can I manage secrets in .tf and .tfstate?

Terraform supports adding an additional file with variables during invocation.

documentation: https://www.terraform.io/intro/getting-started/variables.html#from-a-file

We are using that feature to provide a secrets.tfvars file on each invocation of Terraform. We also use a script to wrap the command so that its invocation is consistent, and all team members avoid having to make the same mistakes. The wrapper synchronizes .tfstate with S3 before an execution, and pushes .tfstate back to S3 at the end. I also hear of people doing the same thing with state stored in Consul, even adding a kind of semaphore in consul to prevent two people from starting Terraform at the same time.

When you avoid setting a default value in a variables.tf file, it forces the user to input the value. It can be either entered manually or using the -var-file command option like described above. Not setting a default on your secrets is a good way to enforce changes that require a change in secrets.

The secrets.tfvars file is a symbolic link to one of the files with secrets which are not stored in version control. We have several, one per environment, like so secrets-prod.tfvars, secrets-dev.tfvars, secrets-stg.tfvars, etc...

An even better practice would be to generate these secrets files during the wrapper script based on data in Vault or some other way to share secrets. Since currently when the format of secrets changes, or secrets themselves, we need to communicate it to the team outside the version control channel - and this doesn't always work well, to be honest. But secrets do change infrequently.


How to move the "All" tab to the right in Jenkins?

One option is to add a new view, remove the "All" view, create a new view, select  "All" and let it start with "z", e.g. "z-all" as the view is sorted alphabetically.

There is an open issue that indicates that the ordering of tabs in Jenkins is alphabetically sorted and cannot be changed at the moment, e.g. change the order manually.


Keep getting permissions error gcloud.container.clusters.get-credentials

I removed the service account and recreated it (with same name and roles). Generated a new key and updated the CI and suddenly it worked. gcloud glitch.


shell script to pass values properties file in java

You can obtain old parameters from the properties file before using sed, e.g.

# get old parameters
$ export oldUsername=$(grep mysql.username $FILE | cut -d'=' -f2)
$ export oldPass=$(grep mysql.pass $FILE | cut -d'=' -f2)

# set new parameters
$ export newUsername=NEW_USERNAME
$ export newPass=NEW_PASS

# change parameters
$ sed -i "s/$oldUsername/$newUsername/g" $FILE
$ sed -i "s/$oldPass/$newPass/g" $FILE


Check a process is running with Salt Stack?

It's a bit hacky, but you can always do something like:

check_process:
  cmd.run:
    name: ps aux | grep '[f]oobar'

The exit code will be non-0 and the state will fail if foobar doesn't exist.


YAML is a very simple, text/human-readable annotation format that can be used to store data.
YAML keeps data stored as a map containing keys and values associated to those keys. This map is in no particular order, so you can reorder it at will. Each pair is in the format KEY: VALUE
When the value is a text string enclose the value with quotes. Some examples below:

KEY1: 'HELLO'
HOST_NUMBER: 1


What steps could be taken to inherit and override some parts of a Chef cookbook?

There are a lot of specifics but the overall pattern we use is "wrap and extend". The general idea is to make a cookbook that depends on the community cookbook, usually named mycompany_originalthing, and then make recipes in that which call include_recipe 'originalthing::whatever' but with either more stuff added before/after or with calls to things like edit_resource to change resources. Avoid edit_resource when possible since it leads to brittle code, but it is there if you need it. You can also use wrapper cookbooks to set attributes, subclass or wrap custom resources, and so on.

For the specific case of "I need to tweak a template in a community recipe" it would look like this:

include_recipe 'original::whatever'

edit_resource!(:template, '/path/to/something') do
  source 'mytemplate.erb'
  cookbook 'mycompany_original'
end

You can find more details about edit_resource and friends at https://coderanger.net/rewind/

With cookbooks based around custom resources instead of recipes, things can get more complex but the specifics depend a lot on the how exactly the cookbook you are extending is written.


How to run experimental Docker features on CircleCI

CircleCI now allows you to use a machine executor which provisions a separate VM for you, with Ubuntu 14.04 and Docker version 17.06.0-ce installed. This allows you to turn on experimental features for the Docker daemon.

You need to use the machine key instead of the docker key, to run your job in a separate virtual machine instead of just a Docker container.

You can only choose 2 images for the machine:

circleci/classic:latest: Ubuntu 14.04 with Docker 17.03.0-ce, or
circleci/classic:edge: Ubuntu 14.04 with Docker 17.06.0-ce - the one with experimental features.

You'll also need to install dependencies on the machine yourself, as it is quite bare. For example, if you need PHP for your tests, you'll need to run sudo apt-get install -y php5.

Here is a sample .circleci/config.yml that builds a Docker image using experimental feature docker build --squash:

  .circleci/config.yml

version: 2
jobs:
  build:
    # Run in a separate virtual machine instead of a Docker container.
    machine:
      enabled: true
      # Use Ubuntu 14.04 with bleeding edge Docker daemon 17.06.0-ce.
      image: circleci/classic:edge
    steps:
      - checkout
      - run:
          command: |
            # Restart Docker with experimental features on.
            sudo sh -c 'echo '\''DOCKER_OPTS="--experimental=true"'\'' >> /etc/default/docker'
            sudo service docker restart

            # Install dependencies for tests etc.
            sudo apt-get update
            sudo apt-get install -y php5

            # Build image with experimental feature --squash.
            docker build --squash -t myuser/myimage .

            # Login and push Docker image to registry.
            docker login -u $DOCKER_USER -p $DOCKER_PASS
            docker push myuser/myimage


How can I build a Linux VM that I can use to test Rest API calls in a lab?

You can test basically any rest api call from postman, a free app that runs perfectly well in windows or Linux (or Mac I’m sure).  You don’t need anything special.

If you want to use Linux specific tools, you can sign up for AWS free tier and make a Linux box in minutes following any of their tutorials (it will be free for 1 year).

You can also use docker on your computer (including on windows).  Docker will be suitable for running most non GUI based or web GUI based Linux apps.  I see a locust image here for example - https://docs.locust.io/en/stable/running-locust-docker.html.


Options to assign a static ip to a service running in docker swarm

On June 30, 2016 this issue was created on GitHub to request the implementation of static IP on docker-swarm services.

Since then multiple replies with thumbs up and down were added.

At the time of writing on December 20, 2019 this issue is open and the feature has not been implemented.

Discussion

After reading all comments I strongly doubt whether static IPs will ever be implemented in docker-swarm. On the other hand it is unclear why this issue has not been closed. If people would like to see that this feature gets implemented, one could create a Pull request or add a thumbs up to create more awareness.


How to pull all alternative tags of a Docker image?

You can list all alternative image using the reg tool.


To me it looks like you are trying to reinvent the wheel. As I understood you would like to restart a container if a health check fails. As you are using docker you should use a orchestration platform in my opinion like Fargate (ECS) or Kubernetes (k8s). I know from the latter that it is possible to define healthchecks, like readiness and health probes. If the health check fails, kubernetes will restart the POD (auto healing).

It could be possible that you would argue that k8s is not an option. If that is the case, you could also have a look to AWS OpsWorks. Although I did not try it myself yet, it is able to perform auto healing as well.


Apparently delegate_to and lininfile or blockinfile do not work properly together, https://github.com/ansible/ansible/issues/28313.

Module blockinfile

It seems redundant lines have not been written as you would expect. Eventually, blockinfile without a marker does the job.

- name: copy the output to a local file
  blockinfile:
    dest: "{{ myshell_output4.stdout }}"
    marker: false
    block: |
      ########################################################
      ###############HOSTNAME:{{ inventory_hostname }}################
      {{ myshell_output3.stdout }}
      ########################END#############################
  delegate_to: localhost

Concatenate files at the end

Alternatively, you can write to different files and concatenate those files after all tasks have run.

- name: copy the output to a local file
  copy:
    content: |
      ########################################################
      ###############HOSTNAME:{{ inventory_hostname }}################
      {{ myshell_output3.stdout }}
      ########################END#############################
    dest: tmp/{{ inventory_hostname }}
  delegate_to: localhost

- name: log files are concatenated
  assemble:
    src: tmp/
    dest: "{{ myshell_output4.stdout }}"
  delegate_to: localhost
  run_once: true

- name: tmp folder is absent
  path:
    src: tmp/
    state: absent
  delegate_to: localhost
  run_once: true

In case your local file myshell_output4.stdout does exist already and you want to append to an  existing file, you assemble the files tmp/* into tmp/assembled and the append them with lineinfile.

- name: append assembled to a local file
  lineinfile:
    dest: "{{ myshell_output4.stdout }}"
    line: "{{ lookup('file', tmp/assembled) }}"
  delegate_to: localhost
  run_once: true


After researching this, i've found an open issue on grafana's github. At the time of writing its still in the backlog.


Memory being abruptly freed up every day at about the same time

I've seen this same "sawtooth" pattern in other systems, in particular a Java-based data tool.  Based on your description, I think you're looking at  .NET garbage collection (assuming this is a .NET app.)  Java and .NET are both memory-managed languages and frameworks that use garbage collection.

A memory leak is typically found in frameworks that lack memory management, or in a program on a memory-managed framework that is overriding or confusing the garbage collector.

The fact that these are your highest-traffic servers makes sense.  You're seeing the .NET framework allocate memory as needed, then the garbage collector kicks in on a regular cycle and reclaims unused memory using the garbage collection algorithms.  Unless you're tracking specific performance issues I don't think this memory usage pattern is a problem.


Application configuration of a complex enterprise system

I highly recommend Consul for configuration management as it is a high reliability, distributed system that provides quite a few and easy to use features with a simple Rest API. Also the community edition is free :), although for a large enterprises there is an enterprise offering.

I am a particular fan of the Key Value store, because it allows me to have separate configurations for my micro-services and back-end services (written in C#/.Net Core and Java respectively) in all three environments (Dev, Beta, Prod).

Whenever I add a new service or application, the configuration for each environment is added to the Key Value store and a simple config client downloads the configuration during the application's bootstrap/loading routines.

You can read a bit more here but in a nutshell:

  Consul has multiple components, but as a whole, it is a tool for
  discovering and configuring services in your infrastructure. It
  provides several key features:

  Service Discovery: Clients of Consul can provide a service, such as
  api or mysql, and other clients can use Consul to discover providers
  of a given service. Using either DNS or HTTP, applications can easily
  find the services they depend upon.

  Health Checking: Consul clients can provide any number of health
  checks, either associated with a given service ("is the webserver
  returning 200 OK"), or with the local node ("is memory utilization
  below 90%"). This information can be used by an operator to monitor
  cluster health, and it is used by the service discovery components to
  route traffic away from unhealthy hosts.

  KV Store: Applications can make use of Consul's hierarchical key/value
  store for any number of purposes, including dynamic configuration,
  feature flagging, coordination, leader election, and more. The simple
  HTTP API makes it easy to use.

  Multi Datacenter: Consul supports multiple datacenters out of the box.
  This means users of Consul do not have to worry about building
  additional layers of abstraction to grow to multiple regions.

  Consul is designed to be friendly to both the DevOps community and
  application developers, making it perfect for modern, elastic
  infrastructures.


How to SSH from Jenkins node to Jenkins node safely?

Yes, storing your ssh keys directly on a build node is a bad practice. Nodes can be replicated, deleted, or given access to from other systems, and you don't want to lose track of what systems have access to your secrets.

You should also not pass them directly into the build job, either as a parameter or as an environment variable. This can cause a huge headache with logging and tracking to make sure your keys aren't inadvertently output to places they shouldn't be. Instead, you should:

Use a centralized secret store (such as Hashicorp Vault) to retrieve the secrets at build run-time.
Use a plugin (such as the Credentials or SSH Credentials Plugin) to reference the keys during the build.


How to use script in a Gitlab bridge job?

One option is to inherit your environment variable from another job. You can create two different jobs, one to create the environment variable, and one to trigger the local CI pipeline.
Once all that is done, you should have something like this:
stages:
  - build
  - deploy

nightlies_env:
  stage: build
  variables:
    BUILD_NAME: "nightly"
  script:
    - BUILD_NAME="${BUILD_NAME}-$(date +"%Y%m%d")"
    - echo "BUILD_NAME=$BUILD_NAME" >> build.env
  artifacts:
    reports:
      dotenv: build.env

nightlies:
  stage: deploy
  variables:
    BUILD_NAME: $BUILD_NAME
  trigger:
    include: build.yml
    strategy: depend


AWS Temporary Permission Granting Tool for Account/Service "Firefights"

Create an IAM role in prod with all the permissions you'd like the devs to have.
When you want devs to have prod access, add the sts:AssumeRole permission to their IAM group.
When it's time to revoke access, remove the sts:AssumeRole permission.

More details in my blog article here.


Add permissions to readers (DevOps Web Portal)

Usually each projects would have teams defined with the team members. So if you have a team defined then add the team to the contributors list.


After getting your site up with docker-compose up there is No practical reason for running docker-compose up again without making any changes, because the container is running.

So docker deamon informs you there is nothing New to do.


I solved this using team-explorer-everywhere. We made a container that clones the code using it edits it an sends it back up. https://github.com/microsoft/team-explorer-everywhere


Best method of retrieving environment information via Salt Stack?

SaltStack provides a method for modifying grains and adding additional information to the grains dictionary in several different ways by either setting them in the /etc/salt/minion config and/or through the /etc/salt/grains file.

For example:
1. add grains to the minion config. Note, simply include the grains key here:

id: minion-07
grains:
  roles:
    - production
  region: nl-amsterdam

add custom grains within the /etc/salt/grains file. Note, add grains without prefacing them with grains:

   roles:
       - production
     region: nl-amsterdam

Alternatively you may set roles via command line shell from master or from salt minion:

  salt 'your minion' grains.setval roles [production, dev, qa]

  salt-call grains.setval roles [production, dev, qa]

For matching grains I usually use Jinja2. Or you may refer to commandline/config file approaches:

  salt -C 'G@roles:production' test.ping

By using sytnax in the top.sls file:

base:
  'roles:production':
    - match: grain
    - salt.master
  'roles:dev':
    - match: grain
    - api-dev

Alternatively you may consider using environments in SaltStack, see here for more details: https://docs.saltstack.com/en/latest/ref/states/top.html#environments

However, roles are more suitable for running specific states, at least We use them in our our company. So up to you what is the best ;)


Why do I need a Nat Gateway for ECS containers using AWSVPC?

While this is completely possible, it would mean that your instance would be accessible from the internet. Public subnet means that your instances are accessible directly  from the outside. Private subnet are not accessible from the outside as instances do not have a publicly  routable IP.
In short, security


Do you use existing automation code (cookbooks/playbooks/dockerfiles/etc) or you write your own from scratch?

My limited experience is indeed that often, you can get a first solution very quickly with pre-made automation code; unfortunately, often there either appear unexpected problems during "first contact", or later during use, especially when you wish to expand the configuration for unusual cases.

So, indeed, I do tend to approach this with an open mind. I will certainly start out with something pre-made, but sometimes my gut feeling tells me to take a look under the hood, and then I'll decide rather spontaneously if I better do it myself, or whether it is a good idea to roll with it...


How to get instance's route table id using CLI?

The route table is associated with VPC, which is associated with the instance.

Given $id shell variable has the instance ID, e.g.

id=i-0xyz # Replace i-0xyz with the real Instance ID.

here is the shell command to get VPCs associated with the instance and assign to $vpcs variable:

vpcs=$(aws ec2 describe-instances --instance-id $id --query 'Reservations[].Instances[].NetworkInterfaces[].VpcId' --output text)

Then to list the route tables associated with VPC, run:

aws ec2 describe-route-tables --filters "Name=vpc-id,Values=$vpcs"

To get just the route ids, add: --query "RouteTables[].RouteTableId[]" parameter.


I would be inclined to use Pipeline Templating as this gives you the ability to define standard templates centrally but leave the SCM field to be defined by the individual job.

  Leveraging templates in this way lets you separate the business logic (what should happen when) of your pipeline from the technical implementation (what’s actually going to happen). The result of this is a CI/CD pipeline that’s proven to be significantly easier to manage when supporting multiple teams simultaneously.

I recommend that you do not have a single pipeline for all repositories as that is going to make it very hard to tell which repository is failing or slow via dashboards and alerts.


Why is Docker best practice to minimise the number of layers in an image?

you should minimize the Layers of an Image for following Reasons:

first, you can only have (afaik) 127 Layers in an Image.
second: yes, you should include severals Steps in a Single run because of the Copy-On-Write functionality:

if you run:
RUN dd if=/dev/zero of=output.dat  bs=1M  count=10
RUN rm -rf output.dat

your image is still 10MB (plus some bytes for the delete Marker)
instead, you should run:
RUN dd if=/dev/zero of=output.dat  bs=1M  count=10 &&\
    rm -rf output.dat

then, your image size is not increasing.
surely, this is not the Usual Use-Case, but, for example from a Real-World-Example:
# install terraform
ENV TF_VERSION="0.12.16"
RUN curl -LO https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip \
  && unzip terraform_${TF_VERSION}_linux_amd64.zip \
  && chmod +x ./terraform \
  && mv ./terraform /usr/local/bin/terraform \
  && rm terraform_${TF_VERSION}_linux_amd64.zip

more informations here: https://docs.docker.com/storage/storagedriver/#the-copy-on-write-cow-strategy


How can successfully pre-verified changes cause regressions that should have been caught?

There's one possibility I can think of, if when the dev are working on their own workstation, with sometimes images baked for virtual box to run on their workstation where your infrastructure doesn't use the exact same image.

The dev will need, while developing a feature, need to add a JVM parameter or whatever change to the middleware early in its work and forget it.

Before commiting, all unit/integration tests run on its workstation works great, as the baked image is shared, it works on every develloper system.

But when going through CI, it fails because the change to the middle-ware wasn't implemented, either because the dev forgot to ask for it, or just because the team in charge of updating the base images/provisioning system didn't had the time or did forget to update the system.

That's a good thing it break in CI, because it tells early before going into production that the system won't work as expected, but sometimes it becomes a hell to find the missing parameter.

This last point advocate to avoid rejecting commits, and just break on CI on a feature branch, thus it won't block anyone else, and let the dev fix the problem early, when the change is needed and prevent this change to be forget in the flow.

FWIW, we did exactly this here, developers had whole access to development machines and releases in Q/A were failing because a parameter change has been forget, we did move to chef to handle the configuration of the middleware (tomcat now) so each needed change to the infrastructure has to be coded somewhere and will be reproduced in all environment.


Why should a developer care about Docker?

Probably not the answer that you're looking for, but an answer nonetheless :)

Learning about docker and its deployment methods could actually be included in the business requirements by making it part of the project or team development environment, just like the code language(s), version control system, compilers, test infrastructure, etc - to work in that team or on that project one needs to know about and use all of these, can't "bring your own" (in most cases).

Things get a bit more complicated if by "a developer" you actually mean the majority of or even entire development team. Pushing a tool in the development environment without actually having any of the developers supporting it will be really tough. Spend the time to create one such supporter first from the team's technical leadership.

Side note: it might also not be necessary for each and every developer in the team to become a docker expert. Pre-established usage recipes wrapped in simple, cheatsheet-ready commands often allow developers to use docker-based solutions without actually knowing too much about their inner workings, which could be fairly acceptable, especially in large teams. Just like being able to contribute code without knowing all the details about how the end-product is being built.


Create dynamic resources using 'for_each' or 'for'

you need to use count, and with it you should be able to access item by index: https://www.terraform.io/docs/configuration/resources.html#the-count-object


Ansible: Run tasks on specific hosts with var/fact from another host

The condition works as expected. For example, the inventory and the playbook below
shell> cat hosts
[managers]
k8s01
k8s02
k8s03

[workers]
worker1
worker2

shell> cat playbook.yml
- hosts: all
  gather_facts: false
  tasks:
    - debug:
        var: inventory_hostname
      when:
        - inventory_hostname != groups.managers.0
        - inventory_hostname in groups.managers

give
shell> ansible-playbook -i hosts playbook.yml

PLAY [all] ****

TASK [debug] ****
skipping: [worker1]
ok: [k8s02] => {
    "inventory_hostname": "k8s02"
}
skipping: [k8s01]
skipping: [worker2]
ok: [k8s03] => {
    "inventory_hostname": "k8s03"
}


Difference between azure devops, azure devops server and TFVC

Q: "What should I do in order to work with Team Foundation Version Control?"
A: TL'DR: Use the TFS command line client, tf.
?  But why use TFVC ?
Azure DevOps is confusingly several products. There is the cloud offering: DevOps Services, formerly known as Visual Studio Team Services, formerly, Visual Studio Online, formerly Team Foundation Service.  There is also an On-Premise offering: Azure DevOps Server, formerly known as Team Foundation Server (now 2020, formerly 2019, 2017, 2015, 2013, 2012, 2010, 2018, 2005). They do share a common REST API, albeit the on-prem lags behind.
Aside from the delay in getting features into the on-prem variant, the questions of choosing on-prem or cloud have nothing to do with choosing TFVC vs Git. It likely depends more on cost, infrastructure and people resources, geographic and network constraints.
You can see most of the Product Announcements from the Program Manager here.
From TFS 2005 to 2012, the version control engine was TFVC, a centralized version control tool. Team Foundation Server 2013 added native support as a centralized Git remote host (similar to Github/Gitlab/BitBucket,but on-prem), albeit in separate team projects. TFS 2015 Update 1 (clumsily) allowed hosting both in the same team project.
But, let's get back to today. Since TFS 2015, Microsoft has put very little effort into improving the TFVC side of their offering. It would seem today, they have essentially abandoned it. Team Explorer Everywhere is no longer maintained. In  2016, they open-sourced it. Admittedly, in  2017 they did reintroduce it after a large outcry. The Eclipse client was last updated for Neon release (2016-06). The tfs-plugin for Jenkins has not had a significant release since Mar 2018. On 2020-11-06 (Nov 6, 2020), the Visual Studio Code plugin will be removed from the VS Code Marketplace; users will have to rely on the tf command line.
The Program Manager's first post detailed TFS Usage within Microsoft. Those were last updated in 2013. A few years later, they are now boasting of having the world's largest Git repo and then acquired GitHub.
The basic question you should be asking is Choosing the right version control for your project? Do you want a Centralized model or a Distributed model of version control?
I have a lot (decades) of experience managing Centralized and Distributed version control systems. As MS documents, each have their strengths, weaknesses and trade-offs. But, why would you choose a centralized version control tool that is provided by a vendor that evidently no longer supports it?

PS: Most people visualize Centralized Version Control like this:
 (from: https://scmquest.com/wp-content/uploads/2014/05/central_vcs.jpg)
and Distributed Version Control like this:
 (from: https://scmquest.com/wp-content/uploads/2014/05/distributed_vcs.jpg)
But that is really just a "disconnected, centralized model". Git allows for very complex distributed workflows, such as show below. You do not get that flexibility using centralized tools. But it can also be a total nightmare to keep track of what's where.

(from: https://howtodoinjava.com/vcs/how-distributed-version-control-system-works/)


Persist Changes to New Docker Image - neo4j

Regarding to Neo4j with Docker and Insallation: Docker

  /data to allow the database to be persisted outside its container.

  /logs to allow access to Neo4j log files.

I understand that we should mount the /data to the host machine so that the new password is persisted as the following example.

docker run \
    --publish=7474:7474 --publish=7687:7687 \
    --volume=/path/to/neo4j/data:/data \
    --volume=/path/to/neo4j/logs:/logs \
    neo4j:latest

Note, the /path/to/neo4j is a physical directory on the host machine.

Edit1

To set password, the Insallation: Docker mentions as

  By default Neo4j requires authentication and requires you to login with neo4j/neo4j at the first connection and set a new password.

  You can set the password for the Docker container directly by specifying --env NEO4J_AUTH=neo4j/<password> in your run directive. Alternatively, you can disable authentication by specifying --env NEO4J_AUTH=none instead.

Then the docker run should be

docker run \
    --publish=7474:7474 --publish=7687:7687 \
    --env NEO4J_AUTH=neo4j/<password> \
    --volume=/path/to/neo4j/data:/data \
    --volume=/path/to/neo4j/logs:/logs \
    neo4j:latest

Anyhow for the /data, I only achieve with mounting it to the host machine.


Vagrant inside a VM to control the VM host

Is the answer as simple as setting up a dual-booted machine? Linux partitions for development tasks and as the VM host, Windows partitions for normal computing use.

Laptop

Windows partition
Linux Partition

Vagrant VM


error validating data: ValidationError(Deployment): unknown field "spec" in

Sure enough, everything and including the last spec: needed to be tabbed to the left once:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      component: web
  template:
    metadata:
      labels:
        component: web
    spec:
      containers:
        - name: client
          image: ldco2016/multi-client
          ports:
            - containerPort: 3000

Unfortunately, you can't tell the difference with how yaml is formatted when pasted here, so it would not have been an easy catch, but if you would be looking at the original file configuration you would have noticed it like I did when I looked a third time.


Understanding contents and workwise of a Terraform plan

Terraform use protobuf to store the plans according to its code

  // writeTfplan serializes the given plan into the protobuf-based format used
  // for the "tfplan" portion of a plan file.

You can find here the code generating the plan file.

So in brief that's a binary file with a serialized version of what to do in a portable format (protocol buffers) for which you find more information here


Where Azure DevOps Services Enviromnent come into play?

You need to enable the multi-stage pipelines preview feature. Here is a guide to enable preview features.


What are the best practices for sharing production Postgres access with developers?

We keep a RO clone of the important DB's.  Any dev can access the basic RO clone if he's in the right dev group vpn,  because the cloudsql instances are deployed in a private service net that peers with our private net and get be gotten to from the outside.  If not they can get added it a group..  Then they just need the ip address and they need the user and password.  There's an ROnly user given access to everything the postgres user gets, just RO.
PII data goes in a second PII DB, which is set up the same, but requires our compliance department to vet the person for pii access
We also have a firm policy of no poking the DB.  Look at it, sure, clone if you have to to determine how to fix a data error, but no raw manhandling.  It's to be done with a (possibly one shot) module off the codes db admin function.


LAMP in docker must be multiple containers or one?

I would go with solution 2: use docker-compose. The idea is that with solution 2 you can easily update mysql for example without having to modify and rebuild your dockerfile. you just have to update the version of the mysql image in the docker compose file.

I found with a quick search on github this repository https://github.com/sprintcube/docker-compose-lamp

Note: for production, use terraform script rather than docker-compose


Is Gradle used to automate deployment of EAR and WAR files to application servers?

If you want just to deploy faster to server(propably not only Glassfish) you can just write code in gradle to do that.
Example taken from here

/**
 *  ~/.gradle/gradle.properties:
 *  glassfishHome=/path/to/glassfish_home
 *
 *  or in Netbeans, right click project, Properties, Manage Build in Tasks, Run
 *  Add line to Arguments: -Dorg.gradle.project.glassfishHome=/path/to/glassfish_home
 *
 *  For more information about Exec tasks see
 *  http://www.gradle.org/docs/current/dsl/org.gradle.api.tasks.Exec.html
 */
task run(dependsOn: 'war', type:Exec) {
    workingDir "${glassfishHome}${File.separator}bin"

    if (System.properties['os.name'].toLowerCase().contains('windows')) {
        commandLine 'cmd', '/c', 'asadmin.bat'
    } else {
        commandLine "./asadmin"
    }

    args "deploy", "--force=true", "${war.archivePath}"
//from: https://www.schuermann.eu/2013/08/03/gradle-glassfish.html
}

In other words just use your asadmin